? H?E? H?ף ?8H?u?L?? ? X ???? H?5?| H??| ? 1?????H? L?c?C?C M???? ???n???b~] ??\?] ??y???X???
?( ??Y???Y?H?5?z H??{ ? ??2???H???V ?
?? uH?=?? ?-? H?r? 1?? H?U?H?L?c?C?C H???`???H?SL?s H?C( M??t&H?5`? L??L??E1?E1??Ƣ H?C(H???.?????a? ??D$0??C? ??D$ ??? ??$L??H??L????w?_???H??X[A\A]A^A_]?UH?? ]? D UH??SPH?=?? H??u?S H??H??? ??? H?5?? ?,? H???
s? uH?=r? ? ? H?a? H??H??[]?ffffff. ? UH??]? ? UH??AWAVAUATSH??? H??I??I??? ? ? T H?E?L?(L?x? ?U H??L??H ?H?? t
L??1?H??I???
D??1???A??M?L??H ?H?? H?]?t
L??1?H???D??1???H?M?H??H)?H??HD?L?L?%?? M??u?R I??H?~? M)??m? L?5~? L??L???
? H9?H?]?v2H?u?H?E?H?H??H??H ?H?? L??x????? 1?H??H???{ H??H?]?L?kL?{ H?C( H??t*H?54? L??L??E1?E1???? H?M?H?C(H???? L?}????n???bxZ ??\?Z ??y???X???
?% ??Y???Y???E?H? R? L?-C? L??H??A?????n???b-Z ??\5Z ??y???X???
K% ??Y???Y???E?L?5?? L??L??A?????n???b?Y ??\?Y ??y???X???
% ??Y???Y?H?5?x H??x ? ??~E???M??????L??H??A??I??L??L??A?ջ I9?vH?5?x H??x ? 1??Z???H?uЉ^L?}??I 1?????L?-M? L??L??A??H?E?L??L??A??H?E?M??H?u?? L?}?H??H?E?H??HE?H?E?H??? H?E?H?? H?E?H??? H?E?I??I??(?E? E1?L?m?L?e? D H?M?I?M)?H;E?LF?H??x???J?8I?V?M?n?I? M??H? ?? t L??H?u?L??E1?E1???I?H???? L?u????n???o
hX ??b???(
lX ??\???y???X???
~# ??Y???Y????h???L??L?m?L???????n???b X ??\%X ??y???X???
;# ??Y???Y????p???L??I??H?]?H??A?????n???b?W ??\?W ??y???X???
?" ??Y???Yѿ H?5?v H??v ??~?h??????p?????z???L??L??A??I??L??H??A??I9?v? H?5?v H??v 1??F???L}?L;}??]?L?u??????? H?5u 1?? ???H?EЉX?É]?I??L?m?L?e??A??????? ??D$0???? ??D$ ??om? ??$H?=!? L????w趇???R???n?????n???b?V ??\?V ??y???X???
?! ??Y???Y?H?5!t H?Rt ? ??t???1?H??? [A\A]A^A_]?UH??AVS?? H???? I?ƿ? ?_O H??H?=? H?8???? ??@??̴ ??@(??ߴ ??@H??? ??@hH?? H??? Hǃ? L??? H????w?"?????t;A??? t)Adž? ? H?5?s H??s 1ۿ 1?????H??[A^]?H?=?s H??$ H?
t ??
1?????f?UH??AWAVAUATSH???H?? H?1? H? H??$ H?5u H?u ? 1??,????L ??W???)D$@??)D$`H?5 ? H?T$@H??$? A? H?D$ H????w??? H???? I??H?D$PL?(L?5? ?=ffffff. ? A? H?|$ H?5?? H?T$@H??$? A??I??H???? L?=? H?d? H?D$01??Nff. ? H?D$HH?r E1?? H??1??M???H?|$ ?K H? ? H? H;?$ ?? L??H?e?[A\A]A^A_]?H?D$H?\$(I??H?|$ ?GK H?;H?t$8A??H??H?t$0A??H?5Wr H??q ? H??1??????L?- ? H?;?? L??A??A?? ???X I???? L????? A?? ???9 I???? L????? A?? ??? I???? L???m? A?? ???? I???? L???N? A?? ???? I???? L???/? A?? ???? I???? L???? A?? ???? I???? L???? A?? ??? I???? L???Җ A?? ???` I???? L????? A?? ???A I???? L????? A?? ???" I???? L???u? A?? ??? I???? L???V? A?? ???? I???? L???7? A?? ???? I???? L???? A?? ???? I???? L????? A?? ???? I???? L???ڕ A?? ??ulI???? L????? A?? ??uQI???? L????? A?? ??u6I???? L????? A?? ??uI???? L???n? A?? ??t!A?????H?5?o H? o ? 1??3???I???? L???2? A?? ???? I???? L???? A?? ??ulI???? L????? A?? ??uQI???? L???ݔ A?? ??u6I???? L??? A?? ??uI???? L????? A?? ??t!A??H???H?50o H?Wn ? 1??l???I???? L???k? A?? ???? I???? L???L? A?? ??ulI???? L???1? A?? ??uQI???? L???? A?? ??u6I???? L????? A?? ??uI???? L????? A?? ??t!A??z???H?5?n H??m ? 1?????I???? L??A?ք?A??? I???? L??A????A??? ?A??? I???? L??A?ք?A??? A??? L?%R L?-kn L??ID?H?55n H? m ? H??1?????A??? L??ID?H?56n ? H??1??????I??H?5I? A?ք?MD?H?55n ? H??L??1??????fALJ? ILJ? ??W???}?? ??}?? ??}? I??H?5i? ??wA?????n???b?L ??\?L ??y???X???^? H?5?m ? H?ڰ?F???H?D$ H?D$(H?x?? ?ID L?-?? H?d? H?|$L??A??H??I??L?%?? H?L$L??H??A??I?GH???? H?|$H???? H?? H?|$H???- H?٫ H?|$L??A??H??I??H?L$L??H??A??I?G8H????? H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I?G@H????? H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I?GHH???{? H?|$H???j H?v? H?|$L??A??H??I??H?L$L??H??A??I?GPH???:? H?|$H???) H?U? H?|$L??A??H??I??H?L$L??H??A??I?GXH????? H?|$H???? H?4? H?|$L??A??H??I??H?L$L??H??A??I?G`H????? H?|$H???? H?? H?|$L??A??H??I??H?L$L??H??A??I?GhH???w? H?|$H???f H?? H?|$L??A??H??I??H?L$L??H??A??I?GpH???6? H?|$H???% H?Ѫ H?|$L??A??H??I??H?L$L??H??A??I?GxH????? H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I??? H???m? H?|$H???\ H?h? H?|$L??A??H??I??H?L$L??H??A??I??? H???)? H?|$H??? H?D? H?|$L??A??H??I??H?L$L??H??A??I??? H???? H?|$H???? H? ? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I??? H???]? H?|$H???L H?ة H?|$L??A??H??I??H?L$L??H??A??I??? H???? H?|$H??? H??? H?|$L??A??H??I??H?L$L??H??A??I??? H???Ռ H?|$H???? H??? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???? H?l? H?|$L??A??H??I??H?L$L??H??A??I??? H???M? H?|$H??u@H?L? H?|$L??A??H??I??H?L$L??H??A??I??? H???
? H?|$H??t]H?5? A??H??H?t$0A??H?5?g H??e E1?? H??1?????H?|$?Ë H??? H? H;?$ ?h?????= H?D$(H? H?D$ ?8 tUH??? H?|$H?5 ? L?5o? A??H??I??H?L$H?5?? H??A??I??? H???L? H?|$H???;????!H?5ah H??d H?
?h ? 1?????H?D$ ?8 tUH?Z? H?|$H?5?? L?5? A??H??I??H?L$H?5z? H??A??I??? H???̊ H?|$H????????!H?5?g H?ud H?
?g ? 1?????H?D$ ?8 tUH??? H?|$H?5 ? L?5o? A??H??I??H?L$H?5?? H??A??I??? H???L? H?|$H???;????!H?5ag H??c H?
?g ? 1?????H?D$ ?8 t[H??? H?|$H?5?? I??L?5? A??H??I??H?L$L?-w? L??H??A??I??? H???Ɖ H?|$H????????6H?5?f H?oc H?
g ? 1??}???L?5?? L?%'? L?-? H?)? H?|$L??A??H??I??H?L$L??H??A??I??? H???J? H?|$H???9???H?? H?|$L??A??H??I??H?L$L??H??A??I?? H???? H?|$H???????H?? H?|$L??A??H??I??H?L$L??H??A??I?? H??? H?|$H???????H??? H?|$L??A??H??I??H?L$L??H??A??I?? H???~? H?|$H???m???H??? H?|$L??A??H??I??H?L$L??H??A??I?? H???:? H?|$H???)???H?u? H?|$L??A??H??I??H?L$L??H??A??I?? H????? H?|$H???????H?Q? H?|$L??A??H??I??H?L$L??H??A??I??( H????? H?|$H???????H?-? H?|$L??A??H??I??H?L$L??H??A??I??0 H???n? H?|$H???]???H? ? H?|$L??A??H??I??H?L$L??H??A??I??8 H???*? H?|$H??????H?? H?|$L??A??H??I??H?L$L??H??A??I??@ H???? H?|$H???????H??? H?|$L??A??H??I??H?L$L??H??A??I??H H????? H?|$H???????H??? H?|$L??A??H??I??H?L$L??H??A??I??P H???^? H?|$H???M???H?y? H?|$L??A??H??I??H?L$L??H??A??I??X H???? H?|$H??? ???H?U? H?|$L??A??H??I??H?L$L??H??A??I??` H???օ H?|$H???????H?1? H?|$L??A??H??I??H?L$L??H??A??I??h H????? H?|$H???????H?
? H?|$L??A??H??I??H?L$L??H??A??I??p H???N? H?|$H???=???H?? H?|$L??A??H??I??H?L$L??H??A??I??x H???
? H?|$H???????H?Ť H?|$L??A??H??I??H?L$L??H??A??I??? H???Ƅ H?|$H???????H??? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???q???H?}? H?|$L??A??H??I??H?L$L??H??A??I??? H???>? H?|$H???-???H?Y? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???????H?5? H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???????H?? H?|$L??A??H??I??H?L$L??H??A??I??? H???r? H?|$H???a???H??? H?|$L??A??H??I??H?L$L??H??A??I??? H???.? H?|$H??? ???H?D$ ?8 tUH??? H?|$H?5?? L?5? A??H??I??H?L$H?5? H??A??I??? H???т H?|$H????????!H?5?_ H?z\ H?
Cb ? 1?舿??H?D$ ?8 t[H?_? H?|$L?%#? L??L?5q? A??H??I??H?L$L?-?? L??H??A??I??? H???K? H?|$H???:????6H?5`_ H??[ H?
?a ? 1?????L?5 ? L?%?? L?-?? H?? H?|$L??A??H??I??H?L$L??H??A??I??? H???ρ H?|$H???????H?ʢ H?|$L??A??H??I??H?L$L??H??A??I??? H????? H?|$H???z???H??? H?|$L??A??H??I??H?L$L??H??A??I??? H???G? H?|$H???6???H?D$ ?8 tUH?x? H?|$H?5?? L?5
? A??H??I??H?L$H?5?? H??A??I??? H???? H?|$H????????!H?5?] H??Z H?
?` ? 1?衽??H?D$ ?8 tUH?? H?|$H?5g H?D$H??H???*????!H?5PD H??@ H?
?K ? 1?????H?D$(?8 tXH?Ɏ H?|$H?5?? L?5?f A??H??I??H?L$H?5i? H??A??I??? H????f H?D$H??H????????!H?5?C H?a@ H?
?K ? 1??o???H?D$(?8 tXH?f? H?|$H?5
? L?5[f A??H??I??H?L$H?5? H??A??I??? H???8f H?D$H??H???$????!H?5JC H??? H?
K ? 1?????H?D$(?8 tXH?? H?|$H?5?? L?5?e A??H??I??H?L$H?5c? H??A??I??? H????e H?D$H??H????????!H?5?B H?[? H?
?J ? 1??i???H?D$(?8 tXH??? H?|$H?5? L?5Ue A??H??I??H?L$H?5? H??A??I??? H???2e H?D$H??H??? ????!H?5DB H??> H?
FJ ? 1?????H?D$(?8 tXH?=? H?|$H?5?? L?5?d A??H??I??H?L$H?5]? H??A??I??? H????d H?D$H??H????????!H?5?A H?U> H?
?I ? 1??c???H?D$(?8 tXH?ڌ H?|$H?5?? L?5Od A??H??I??H?L$H?5ڨ H??A??I??? H???,d H?D$H??H???????!H?5>A H??= H?
nI ? 1?????H?D$(?8 tXH?w? H?|$H?5{? L?5?c A??H??I??H?L$H?5W? H??A??I??? H????c H?D$H??H????????!H?5?@ H?O= H?
I ? 1??]???H?D$(?8 tXH?? H?|$H?5?? L?5Ic A??H??I??H?L$H?5ԧ H??A??I??? H???&c H?D$H??H???????!H?58@ H??< H?
?H ? 1??ڟ??H?D$(?8 tXH??? H?|$H?5u? L?5?b A??H??I??H?L$H?5Q? H??A??I??? H????b H?D$H??H????????!H?5?? H?I< H?
2H ? 1??W???H?D$(?8 tXH?N? H?|$H?5? L?5Cb A??H??I??H?L$H?5Φ H??A??I??? H??? b H?D$H??H??? ????!H?52? H??; H?
?G ? 1??Ԟ??H?D$(?8 tXH?? H?|$H?5o? L?5?a A??H??I??H?L$H?5K? H??A??I??? H????a H?D$H??H????????!H?5?> H?C; H?
\G ? 1??Q???H?D$(?8 tXH??? H?|$H?5? L?5=a A??H??I??H?L$H?5ȥ H??A??I??? H???a H?D$H??H???????!H?5,> H??: H?
?F ? 1??Ν??H?D$(?8 tXH?%? H?|$H?5i? L?5?` A??H??I??H?L$H?5E? H??A??I??? H????` H?D$H??H????????!H?5?= H?=: H?
?F ? 1??K???H?D$(?8 tXH? H?|$H?5? L?57` A??H??I??H?L$H?5¤ H??A??I??? H???` H?D$H??H??? ????!H?5&= H??9 H?
F ? 1??Ȝ??H?D$(?8 tXH?_? H?|$H?5c? L?5?_ A??H??I??H?L$H?5?? H??A??I??? H????_ H?D$H??H???}????!H?5?< H?79 H?
?E ? 1??E???H?D$(?8 tXH??? H?|$H?5? L?51_ A??H??I??H?L$H?5?? H??A??I??? H???_ H?D$H??H????????!H?5 < H??8 H?
HE ? 1????H?D$(?8 tXH??? H?|$H?5]? L?5?^ A??H??I??H?L$H?59? H??A??I?? H????^ H?D$H??H???w????!H?5?; H?18 H?
?D ? 1??????H?D$(?8 tXH?6? H?|$H?5ڢ L?5+^ A??H??I??H?L$H?5?? H??A??I?? H???^ H?D$H??H????????!H?5; H??7 H?
uD ? 1?輚??H?D$(?8 tXH?Ӈ H?|$H?5W? L?5?] A??H??I??H?L$H?53? H??A??I?? H????] H?D$H??H???q????!H?5?: H?+7 H?
D ? 1??9???H?D$(?8 tXH?p? H?|$H?5ԡ L?5%] A??H??I??H?L$H?5?? H??A??I?? H???] H?D$H??H????????!H?5: H??6 H?
?C ? 1?趙??H?D$(?8 tXH?
? H?|$H?5Q? L?5?\ A??H??I??H?L$H?5-? H??A??I?? H???\ H?D$H??H???k????!H?5?9 H?%6 H?
:C ? 1??3???H?D$(?8 tXH??? H?|$H?5Π L?5 \ A??H??I??H?L$H?5?? H??A??I??( H????[ H?D$H??H????????!H?59 H??5 H?
?B ? 1?谘??H?D$(?8 tXH?G? H?|$H?5K? L?5?[ A??H??I??H?L$H?5'? H??A??I??0 H???y[ H?D$H??H???e????!H?5?8 H? 5 H?
hB ? 1??-???H?D$(?8 tXH?? H?|$H?5ȟ L?5[ A??H??I??H?L$H?5?? H??A??I??8 H????Z H?D$H??H????????!H?58 H??4 H?
?A ? 1?誗??H?D$(?8 tXH??? H?|$H?5E? L?5?Z A??H??I??H?L$H?5!? H??A??I??@ H???sZ H?D$H??H???_????!H?5?7 H?4 H?
?A ? 1??'???H?D$(?8 tXH? ? H?|$H?5 L?5Z A??H??I??H?L$H?5?? H??A??I??H H????Y H?D$H??H????????!H?57 H??3 H?
-A ? 1?褖??H?D$(?8 tXH??? H?|$H?5?? L?5?Y A??H??I??H?L$H?5? H??A??I??P H???mY H?D$H??H???Y????!H?56 H?3 H?
?@ ? 1??!???H?D$(?8 tXH?X? H?|$H?5?? L?5
Y A??H??I??H?L$H?5?? H??A??I??X H????X H?D$H??H????????!H?5?5 H??2 H?
^@ ? 1?螕??H?D$(?8 tXH??? H?|$H?59? L?5?X A??H??I??H?L$H?5? H??A??I??` H???gX H?D$H??H???S????!H?5y5 H?
2 H?
?? ? 1?????H?D$(?8 tXH??? H?|$H?5?? L?5X A??H??I??H?L$H?5?? H??A??I??h H????W H?D$H??H????????!H?5?4 H??1 H?
?? ? 1?蘔??H?D$(?8 tXH?/? H?|$H?53? L?5?W A??H??I??H?L$H?5? H??A??I??p H???aW H?D$H??H???M????!H?5s4 H?1 H?
)? ? 1?????H?D$(?8 tXH?̂ H?|$H?5?? L?5W A??H??I??H?L$H?5?? H??A??I??x H????V H?D$H??H????????!H?5?3 H??0 H?
?> ? 1?蒓??H?D$(?8 tXH?i? H?|$H?5-? L?5~V A??H??I??H?L$H?5 ? H??A??I??? H???[V H?D$H??H???G????!H?5m3 H?0 H?
Y> ? 1?????H?D$(?8 tXH?? H?|$H?5?? L?5?U A??H??I??H?L$H?5?? H??A??I??? H????U H?D$H??H????????!H?5?2 H?~/ H?
?= ? 1?茒??H?D$(?8 t^H??? H?|$L?%'? L??L?5uU A??H??I??H?L$L?- ? L??H??A??I??? H???OU H?D$H??H???;????6H?5a2 H??. H?
?= ? 1?????L?5 U L?%?? L?-?? H?/? H?|$L??A??H??I??H?L$L??H??A??I??? H????T H?|$H???????H? ? H?|$L??A??H??I??H?L$L??H??A??I??? H????T H?D$H??H???x???H?? H?|$L??A??H??I??H?L$L??H??A??I??? H???ET H?D$H??H???1???H??? H?|$L??A??H??I??H?L$L??H??A??I??? H????S H?D$H??H???????H??? H?|$L??A??H??I??H?L$L??H??A??I??? H????S H?D$H??H???????H?o? H?|$L??A??H??I??H?L$L??H??A??I??? H???pS H?D$H??H???\???H?H? H?|$L??A??H??I??H?L$L??H??A??I??? H???)S H?D$H??H??????H?!? H?|$L??A??H??I??H?L$L??H??A??I??? H????R H?D$H??H???????H?? H?|$L??A??H??I??H?L$L??H??A??I??? H????R H?D$H??H???????H?? H?|$L??A??H??I??H?L$L??H??A??I??? H???TR H?D$H??H???@???H?? H?|$L??A??H??I??H?L$L??H??A??I??? H???
R H?D$H??H???????H?? H?|$L??A??H??I??H?L$L??H??A??I??? H????Q H?D$H??H???????H?^ H?|$L??A??H??I??H?L$L??H??A??I??? H???Q H?D$H??H???k???H?D$(?8 tXH?- H?|$H?5? L?5BQ A??H??I??H?L$H?5͕ H??A??I?? H??? Q H?D$H??H??? ????!H?51. H??* H?
?: ? 1??Ӎ??H?D$(?8 tXH??~ H?|$H?5n? L?5?P A??H??I??H?L$H?5J? H??A??I?? H????P H?D$H??H????????!H?5?- H?B* H?
": ? 1??P???H?D$(?8 tXH?g~ H?|$H?5? L?5} H?|$L?-b? L??L?5?N A??H??I??H?L$L?%;? L??H??A??I??( H????N H?D$H??H???v????6H?5?+ H?0( H?
?8 ? 1??>???L?5GN L?-? L?%ْ H??| H?|$L??A??H??I??H?L$L??H??A??I??8 H??? N H?|$H???????H??| H?|$L??A??H??I??H?L$L??H??A??I??0 H????M H?D$H??H???????H?| H?|$L??A??H??I??H?L$L??H??A??I??@ H????M H?D$H??H???l???H?X| H?|$L??A??H??I??H?L$L??H??A??I??H H???9M H?D$H??H???%???H?1| H?|$L??A??H??I??H?L$L??H??A??I??P H????L H?D$H??H???????H?
| H?|$L??A??H??I??H?L$L??H??A??I??X H????L H?D$H??H???????H??{ H?|$L??A??H??I??H?L$L??H??A??I??` H???dL H?D$H??H???P???H??{ H?|$L??A??H??I??H?L$L??H??A??I??h H??? L H?D$H??H??? ???H??{ H?|$L??A??H??I??H?L$L??H??A??I??p H????K H?D$H??H???¿??H?n{ H?|$L??A??H??I??H?L$L??H??A??I??x H????K H?D$H??H???{???H?G{ H?|$L??A??H??I??H?L$L??H??A??I??? H???HK H?D$H??H???4???H? { H?|$L??A??H??I??H?L$L??H??A??I??? H???K H?D$H??H???????H??z H?|$L??A??H??I??H?L$L??H??A??I??? H????J H?D$H??H???????H??z H?|$L??A??H??I??H?L$L??H??A??I??? H???sJ H?D$H??H???_???H??z H?|$L??A??H??I??H?L$L??H??A??I??? H???,J H?D$H??H??????H??z H?|$L??A??H??I??H?L$L??H??A??I??? H????I H?|$H??? ???H?5؍ H? ?I ??H??H?t$0???ʽ??H??tUH??H??H?5m? ]???1?? @ UH??AWAVSPH??t? 詎????t I??? L??( H??0 H??[A^A_]?H?=d! H?:? H?
?! ??
1??4??? @ UH??AVSH??tC??I??H??H?5Տ ?@?????t+I??? H?8Hc?H??? H?5)? ??H ????[A^]?H?=? H??? H?
;! ??
1??????UH??SPH??t+H??H??H?5h? ?Ӎ????tH??? ƀ? H??[]?H?=? H?o? H?
? ??
1??i???f ? UH??AVS?Ĭ??H???? I?ƿ? ??? H??H?=? H?8???` ??@???` ??@(???` ??@H???` ??@hH??` H??? Hǃ? L??? H????w??????t;A??? t)Adž? ? H?5? H?? 1ۿ 1??{???H??[A^]?H?=? H?x? H?
? ??
1??r???f?UH??H?? ]? UH??AWAVATSH?_X?{ ~1L?s(E1?L?%5G ffff. ? I?>A??I??HcCI??I9?|??
T? t+?{ tH??F ?8H?3H?S?*? H??[A\A^A_]?? H?=(? ??F H?? ?{ u??? ? UH??H?GXH? ]? UH??H? H??H??L??]?? f D UH??H? H??H??L??]?? f D UH??AWAVATSI??I??H?~?2???Ä?t$M?? M?? L??????L??L??H???9? ??[A\A^A_]ÐUH??H?GXH?8H?P]? ? ff. ? UH??H?? ]? UH??AWAVATSH??L??? H?5(1 H?31 ? 1?衂??A? L?=?E @ K?A??I?~?
? L???D? H??[A\A^A_]?4? ffffff. ? UH??H??? ]? UH??AWAVAUATSH??hI??L??? Mc?$? D??H?E??L? H?E?M??$? A?G=? ?? L?A??$? )?A??$? A??$? ??ș??A??$? A??$? ??L?u??M??O AƄ$? A??$? ?7 H?=? H?5?? H? ?D ??I?$H?5 ? H????I??$? H?=? ?? I??I??$? H?56? H????H?5:? ? L????L?5?? H?=?? H?5c? H??t 1???H?5Y? L??H????H?5? L??H????HDžx??? H?=h? H?5?? ??H?5?? H??x???H??L???ӄ?t I??$? H?5?? ??C AƄ$? ?9H??x???H?5?? ??H?5^? H????H?5!/ H?C/ ? H??1??\???L?u??M?H??B H??x???? ?H?E?H?? H?E?H?C\ H?E?L?e??M?L?}?H??x???I??$? I?|$H? ?? L?-
C H??A??K???? H?5-? H??A??I??$? L???W?}? ~PI?|$H??A??I??$? H??H?5?? A?Ճ}?t*I?|$H??A??I??$? H??H?5І A?Ճ}??? I?t$I??$? H?]?H????? I???? H?5?? H??A??H?5~? H??A??H???? ?}? ?7 H?%? H?E?E1??L??H?u?A??f?L9}?? L??O???? L??H?5Q? A??L??H?5 ? A??H???? L?{L;}?}?M???? M??t?L??H?5? A??H??u?I??$( H??t?I??$0 ?Є??s???H?5?- H?Y- A? ? ??1??m~??? I??H?5X- H?/- ? H?M?I??1??D~??A?????I???? H?5}? H??A??H?5?? H??A??H?5? H??A??H?5.- ? H??1???}???DE1??}? u;A??$? t0I??$? H?5? A??H?=X? H?5? A??H?5?? H??A??H?}???? D??H??h[A\A]A^A_]?H?E?L?x?E1??f ? I??M9??????I?|$H??A??K???? I??$( u?H??H?5?? A????I??H?57, H?, ? ??I??1??%}??I??u=H?5h? L??A??H?5{? H??A??H?5ރ H??A??H?5, ? H??1???|??H?}?A??????&??? UH??H??? ]?Ϗ ffffff. ? UH??H?5???H9??]?ffff. ? UH??AWAVAUATSH??8I??H??H?G D??? ??? ?MЋ?? ?M???? ?M?Mc?J???? H?5I? H?}??G? D??I??E1?D9?L?e?~9?M?A??E??D?u?E???A?D$???M?9?M?D9?D?A?E??E9?|? D?e?E9??? H? ? H?E?H??? H?E?H??? H?E?L?}?? ? A??E9?to?{0 tEL?=? H?{(D???߬??H???'???? L??L?}?H?u?H??L?5}> A??L??H?u?H??A??H?{ D??L???n ?{0 t?L??H?u??K> ?H?5j? L???9> ?}?| H?C H??( tH??8[A\A]A^A_]?H?5?? H?}?H??8[A\A]A^A_]?%?= ? UH??AWAVAUATSH??
I??A??H?}?H??? ????H??H??? H?E?L??? L??? H??貌??????r ?CP?H߃???r ????r H?}?H???w? ???d} H?}?H???p H?OH??H??`???H??X???H?OH??H??(???H?M?H?O H??H?????H??p???H?O(H??H?? ???H??0???H?G0H?????H?O8H??H??@???H??8???H?G@H?????H?GHH??????M???i I?OH??H?????H??????I?OH??H??P???H??x???I?O H??H??????H??@???I?O(H?? ???I?G0H??????I?G8H??????I?G@H??????I?GHH??H???M??H???????f I?L$H??H??????H??????I?L$H??H??????H?????I?D$0H??????I?D$8H??0???I?D$@H??????I?D$H?W HDžX??? H?E? HDžp??? HDž0??? HDž??? HDž8??? HDž??? 1?H??(???1?H?? ???1?H??@???1?H?????1?H??`???1?H??????M???????HDž???? HDžx??? HDž@??? HDž ??? HDž???? HDž???? HDž???? 1?1?H??????1?H?????1?H??P???1?H??H???M??H???????????HDž???? HDž??? HDž???? HDž0??? HDž???? 1?H??????1?H??????1?H??????H?SH??P???H?CH??????H??H???H?C H??????H?????H?C(H??????H??????H?C0H??????H?C8H??????H?C@H??????H?CHH???????$ ?$ H??t?M??tA?H????????????H??(??????|???H??? ?ԁ??L?U?I??? H??ID?H?IH?IX?q??~8I?? H??H?x?????;l L??? H?E? M??L??`?????' H?u?L???{ A???xp H?u?H?NH??H???I9O?p H??????I?G8?d' L????t??????k H?E?H??? H?5em L?=n( L?e?L??A??L?55m L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H????o??H??@???HDžH??? HDžP??? HDžX??? HDž`??? HDžh??? H?5l H??h???H?D$(???X?????D$H??P???H?D$???@?????\ A?<$?.k M?p L???s????L??`????4k L???s?????Ek ?????? L???????Rk L??????H?]?H?;H?5?k ?L' H????? H?H??A9??=k H?;H?5pl ?? ? ' ??L???????? I??@?? D???? ?? E9??? H?E?? H?? ?2q H?
?s Hc?H?????@????/q H?E?H? ??X I?X0A?8?Go??H9???j H?E?H??? H?5|k L?5?& L?}?L??A??H? Lk L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? "k H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A?
L??H??A??H??x???? A? L??H??A??H??@???? A? L??H??A??H?? ???? A?
L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??H???? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H?E?H??p???H??0???H??????H??????H??????HDž???? HDž???? HDž???? H?5?g H??????H?D$(?????????D$H??????H?D$?????????G L???so?????xg H?E?H??? H?5 h L?=# L?e?L??A??L?5?g L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H???~j??H?????HDž??? HDž ??? HDž(??? HDž0??? HDž8??? H?5?f H??8???H?D$(???(?????D$H?? ???H?D$???????sW L???n??????f H?E?H??? H?5$g L?=-" L?e?L??A??L?5?f L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H???i??H??p???HDžx??? HDž???? HDž???? HDž???? HDž???? H?5?e H??????H?D$(?????????D$H??????H?D$???p????V H??(??????-j ??H?
?p Hc?H???H?E?H??X?_, ??`?????e L???mm????L?}?H?M???e ?CX?E??CT??h???H??? H?5?e L?5? L??A??H? ?e L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? ?e H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??h???? A? L??H??A??H?U?? A? L??H??A??H?5?d ?? L??1?A??Hc?h???H?????HDž??? HDž ??? HDž(??? HDž0??? HDž8??? H?5?c H??8???H?D$(???(?????D$H?? ???H?D$???????YC H?E?H??? ?CT?E?H?5-d L?56 L?}?L??A??H? ?c L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??H? ?c H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A?
L??H??A??H??????? A? L??H??A??H??x???? A? L??H??A??H??@???? A?
L??H??A??H?? ???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??H???? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H?U?? A? L??H??A??H??P???H= ? HL?H??H???H?????H??????H??p???H??x???H??????Hc?H??????HDž???? HDž???? H?5?_ H??????H?D$(?????????D$H??????H?D$???p????? L??`???A???_` L???&i?????p` ?CT?E?A? I??? L?e?H?E?|IL??A?@ I?? |7A?? H?? |(A? H?? |E1?H?? A??I?? I?? H??? H?5?_ L?= L??A??H? ?_ L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? ?_ H??X???? A? L??H??A??H??8???? A? L??H??A??H?U?? A? L??H??A??H?5_ ?? L??1?A??H?}??)b??H??????HDž???? HDž???? L??????HDž ??? HDž??? H?58^ H?????H?D$(?????????D$H??????H?D$????????O A? H?? ??c H?
Yf Hc?H???H?E?H ??; A?8??^ A?? ??^ ?????^ ?KT?M??KX??h????K\?? ????K`???????Kd???????Kh??X????sl1???@??H?WH?OM?L?I?|? ???? A?w??h???A?w??p???E?hLcS ?? A?8 ??^ ?CT?E??KX??h???A??A?? A?A??H?E?H??? H?5?] L?5? L?e?L??A??H? ?] L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? }] H??????? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??A?? ? AL?H??`???H??`???HDžh??? HDžp??? H?H??x???HDž???? HDž???? H?5\ H??????H?D$(???x?????D$H??p???H?D$???`????o L??M???We?????F] ?CT?E?H??`???H= A? LL?H?E?H??? H?5p\ L?5y L?e?L??A??H? @\ L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? \ H??X???? A? L??H??A??H??8???? A? L??H??A??H?U?? A? L??H??A??Ic?H?? H???H?5?[ L??1?A??L???^??H??@???HDžH??? HDžP??? H??X???HDž`??? HDžh??? H?5?Z H??h???H?D$(???X?????D$H??P???H?D$???@?????$L??A???kK A?8 ??[ ?;L??`????\ L????]??I?ƋKT? H??H?L9?|?H?E???L?m??Z ????] H?M?H??? ?M A?8 ??[ H?E?H??? H?5?Z L?5? L?}?L??A??H? ?Z L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? \Z H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??P???? A?
L??H??A??H??H???? A? L??H??A??H?????? A? L??H??A??H??????? A?
L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??P???H= ? HL?H??H???H?????H??????H?? ???H?????H?????Hc?H?????HDž ??? HDž(??? H?5?W H??(???H?D$(????????D$H?????H?D$??? ????97 ?; ?GY ?CT?E??C\??h???H?E?H??? H?5?W L?5 L?}?L??A??H?5?W L??H?U?H?M?E1?A??H? ?W H??P???? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??H??P???H= ? HL?HDž0??? HDž8??? HDž@??? Hc?H??H???HDžP??? HDžX??? H?5OV H??X???H?D$(???H?????D$H??@???H?D$???0???? 6 A?8 ?.X ?CT?E?H?E?H??? H?5?V L?=? L?e?L??A??L?5?V L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H?5pV H?U?? A? L??A??H???#Y??H??????HDž???? HDž???? HDž???? HDž???? HDž???? H?5NU H??????H?D$(?????????D$H??????H?D$????????F L??`???L;?????DW L??????L??H?????H ?H?? H?? ????Q L??H?H??H???J ?CT?E?1???`?????H?M?H???? H?5}U L?5? L?}?L??A??H? MU L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??H? #U H??X???? A? L??H??A??H?U?? A? L??H??A??H?U?? A? L??H??A??H??X???H?E????n H??p???H??`???H??h???H??p???HDžx??? HDž???? HDž???? H?5?S H??????H?D$(???x?????D$H??p???H?D$???`???? H?????H;??????U H??`???H= ? HLȋCT?E??CX??h????C\?Cd?? ????Ch???????Cl???????Cp??X????Ct??h????Cx??p????C|??8???A? H??`????! ???? ????W H?E?H? ? A?8 ?VU ???*????????*H??^????*???????E????*@??^????*????????h??????*@ ??^????*???????? ??????*@(??^????????H?E?H??? H?5,S L?=5 L?e?L??H??A??L?5?R L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??L?5?R H??X???? A? L??L??A??H?U?? A? L??L??A??H??p???? A? L??L??A??H??0???? A? L??L??A??H?????? A? L??L??A??H??8???? A? L??L??A??H?????? A? L??L??A??H??????? A? L??L??A??H??P???? A?
L??L??A??H??H???? A? L??L??A??H?????? A? L??L??A??H??????? A?
L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H?U?? A? L??L??A??H??h???? A? L??L??A??H?? ???? A? L??L??A??H??????? A? L??L??A??L?5JP H??L??A??Hc?H??P???H9?} H??L???? H??H???H?????H??????H??????H??????H??????H?H??????HDž???? HDž???? H?5~O H??????H?D$(?????????D$H??????H?D$???????? I??(?
I??8?I??H?E? ?? H??(????? ? U ??H?
nX Hc?H???H?E?H? ?4 1???Lc?H??(?????t3???ZK ?;? H??(????? ?&K ??H?
P Hc?H????????? ?4K H??? ?j% 1?H??????1?H?u?H??h????CT?? ????KX???????S\????????n???W???.???n???M?u{??n???^???? ????F ???*??ڳ ??y
? ???,ȸ ????X?????y&? ??WE????*????(?????^??8? ???h?????(E???YB? ??^?(????? ???p???H?}???A???? ?(????? I??? ?xH H?E?H( H?H?5'E L?=0 H?]?H??H?U?A??L?%?D H??L??H?U?L?u?L??E1?A??A? H??L??H??????H??????A??A? H??L??H??`???L??A??H??????H???? H?M?A? H??L?=?D L???? I?F?L??H???I H?
?K Hc?H???H?E?H ?/???M??0 H?5RD L?5[? L?m?L??L??A??L?% D L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H? ?C H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??P???? A?
L??H??A??H??H???? A? L??H??A??H?????? A? L??H??A??H??????? A?
L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??M??H??A??H?5?A L??I??A??Hc?H??X???H9?} L??L???:? H?M?H??p???H??0???H??0???H??8???H??@???H?H??H???HDžP??? HDžX??? H?5
A H??X???H?D$(???H?????D$H??@???H?D$???0?????$L?m?L????? L?}?H?5?A H?E?L?xL?5?? L??L??A??L??L??H?U?H?M?E1?A??A? L??L??H??????H??????A??A? L??L??H?U?H?M?A??H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??L?? ???? A?
L??H??L??A??H??????? A? L??H??A??H??x???? A? L??H??A??H??@???? A?
L??H??A??H?? ???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??H???? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A?ֹ A? L??H??H?U?A?ֹ A? L??H??H??h???A?ֹ A? L??H??L??A??H??????? A? L??H??A??L??H?]?H??A??Hc?H??X???H9?} L??H????? H??x???H??@???H?? ???H??`???H??h???H??p???H?H??x???HDž???? HDž???? H?5Q= H??????H?D$(???x?????D$H??p???H?D$???`?????$L????? ?. A? H??L?=?= L??H?U?L????? I??A? H??L??H?U?H?M?L?=?? A??L?5?= H?U?? A? H??L??A??H??p???? A? H??L??A??H??0???? A? H??L??A??H??8???? A? H??L??A??H?????? A? H??L??A??H??????? A?
H??L??A??H??x???? A? H??L??A??H??@???? A? H??L??A??H?? ???? A?
H??L??A??H??????? A? H??L??A??H??????? A? H??L??A??H??H???? A? H??L??A??H??0???? A? H??L??A??H??????? A? H??L??A??H??????? A? H??L??A??H??h???? A? H??L??A??H??H???? A? H??L??A??H?????? A? H??L??A??H?? ???? A? H??L??A??H??????? A? H??L??A??H??h???? A? H??L??A??H??p???? A? H??L??A??H??X???? A? H??L??A??H??????? A? H??L??A?׀?(??? L??`???tUH??x???H?X H??HI?H??H?5n: L?u?L????? H?H H??HI?H??H9??% H??x???H? ;: ?% ? H?5": L?m?I??H? ?H??L?H??I?} I????? L??K? ?H9?v?L??`???? H?}??, M??I???M?I??H??x???L?x H??LI?I??L?%?9 H?]?H??L???2? H?H H??HI?H??I9??x H??x????x H?E?H? ?? H?E?H??`?H?E?H??p?H?E?H??hL?e?H?H?5?9 L?=?? L??H??A??L?5?9 L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??L?5l9 H??X???? A? L??L??A??H?U?? A? L??L??A??H??p???? A? L??L??A??H??0???? A? L??L??A??H?????? A? L??L??A??H??8???? A? L??L??A??H?????? A? L??L??A??H??????? A? L??L??A??H??P???? A?
L??L??A??H??H???? A? L??L??A??H?????? A? L??L??A??H??????? A?
L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??L?5P7 H??L??A??Hc?H??P???H9?} H??L????? H??H???H?????H??????H?? ???H?????H?????H?H?????HDž ??? HDž(??? H?5?6 H??(???H?D$(????????D$H?????H?D$??? ????? H?E?H? ?
H?E?H? H?H?57 L?e?L???? L?=?6 L??L??H?]?H??L?u?L??E1???? A? L??L??L?=?? H??????H??????A??H??`???A? L??H??tH? x6 H??L???H?5i6 H??H??L????? A? L??H??H?U?H?M?A??H? D6 H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H??0???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A?
L??H??A??H??????? A? L??H??A??H??P???? A? L??H??A??H??H???? A?
L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??H?? ???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??X???? A? L??H??A??H??h???? A? L??H??A??H??p???? A? L??H??A??H??8???? A? L??H??A??H?E?H??p???H??0???H??p???H??x???H??????Hc?`???H??????HDž???? HDž???? H?5?2 H??????H?D$(?????????D$H??????H?D$???p????_# I??0?
I??@?I??P??E?M?6H?53 L?- ? L?e?L??L??A??L?=?2 L??L??H?U?H?M?E1?A??A? L??L??H??????H??????A??A? L??L??H?U?H?M?A??L?=?2 H??X???? A? L??L??A??H?U?? A? L??L??A??H??p???? A? L??L??A??H??0???? A? L??L??A??H?????? A? L??L??A??H??8???? A? L??L??A??H?????? A? L??L??A??H??????? A?
L??L??A??H??????? A? L??L??A??H??x???? A? L??L??A??H??@???? A?
L??L??A??H?? ???? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??H???? A? L??L??A??H??P???? A? L??L??A??H??H???? A? L??L??A??H?????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H??????? A? L??L??A??H?U?? A? L??L??A??H??h???? A? L??L??A?Հ}? ?? H???2??H?HH??HI?H??H??????HDž???? HDž???? HDž???? HDž???? HDž???? H?5?. H??????H?D$(?????????D$H??????H?D$?????????$L???_? ? H? ?. L??H???G? Hc?H??P???H9?} L??H???,? H?M?H??p???H??0???H??????H??????H??????H?H??????HDž???? HDž???? H?5?- H??????H?D$(?????????D$H??????H?D$???????? L??H? . H????? ? H??@?? H??x???H??H?X H??HI?H??L?u?L???d? H?H H??HI?H??H9??) H??x????- L?=9? ?3 H?E?H? L?0?CT?E??[X??h???L??M????0??I?L$H??H ?H?? tH?H???1???????*??? ??y
? ???,ȸ ???? ?????n???M???yX'? ???????*???M???^??=? ?????????(E???YG? ??^E?? ? ???????H?5f- L?%o? L??L??A??H?57- L??H?]?H??L?u?L??E1?A??H??????A? L??H??tH? - H??H???????H?5?, H??H??L???
? A? L??H??H?U?H?M?A??H? ?, H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H?? ???? A?
L??H??A??H?5#, ?? L??1?A??H??p???H?E?H??0???H?? ???HDž??? HDž??? D??H?????HDž ??? HDž(??? H?5?* H??(???H?D$(????????D$H?????H?D$??? ?????$L??A??? I??X ?????I??` ?????I??h ?????I??p ?????I??x ????I??8 ????I??P ????H?E?H??? H?5?* L?=? L?e?L??A??L?5?* L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H???o-??H?? ???HDž(??? HDž0??? HDž8??? HDž@??? HDžH??? H?5?) H??H???H?D$(???8?????D$H??0???H?D$??? ????d H?E?H??? H?5%* L?=.? L?e?L??A??L?5?) L??L??H?U?H?M?E1?A??A? L??L??H?U?H?M?A??H???,??H??P???HDžX??? HDž`??? HDžh??? HDžp??? HDžx??? H?5?( H??x???H?D$(???h?????D$H??`???H?D$???P???? H???,??L?xH??LI?I??1ɨ??LE?H?E?H??Ȩ H?5-) L?56? L?e?L??A??H? ?( L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??L??????HDž???? HDž???? HDž???? HDž???? HDž???? H?5?' H??????H?D$(?????????D$H??????H?D$????????/???H???*+??L?xH??LI?I??1ɨ??LE?H?E?H??ȸ H?5=( L?5F? L?e?L??A??H?
( L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??L??????HDž???? HDž???? HDž???? HDž???? HDž???? H?5?& H??????H?D$(?????????D$H??????H?D$????????????H???:*??L?xH??LI?I??1ɨ??LE?H?E?H???? H?5M' L?5V? L?e?L??A??H? ' L??H??H?U?H?M?E1?A??A? L??H??H?U?H?M?A??L??????HDž???? HDž???? HDž???? HDž ??? HDž??? H?5?% H?????H?D$(?????????D$H??????H?D$????????O???H??L????? H?P H??HI?H??L??I9?L?=z? ?? H??x???L?x H??LI?I??H?]?H??L?%?% L???G? H?H H??HI?H??I9??^ H??x????^ L??H?5?% ?? H?H H??HI?H??? H??H?H9?~?H??H??H????????H!?H??H??H?L?L??I?M?H?E?H?8H?5,% ??? I9?? ) I??I???H?5?% L??L??1?A??H?E?H??p???H??0???H?? ???H??(???H??0???HDž8??? H??@???HDžH??? H?5h$ H??H???H?D$(???8?????D$H??0???H?D$??? ?????$L???2 H?E?H ????H?E?H ????H?E?H ????H?E?H ????H??L????? H?H H??HI?H??L?=?? H??L??? ?? H??x???H?X H??HI?H??H?}?L?%?# L???s? H?H H??HI?H??H9?} H??x????
H?}?L???I? H?H H??HI?H??L??I9?L???* H??x???H?X H??HI?H??L?m?L??L?%# L????? H?H H??HI?H??H9??? H??x????? H?E?H ? H?E?H ?? H?E?H ?? H?E?H( ?? H?E?H0 ?? H?E?H8 ? H?E?H@ ? H?E?HH ? H?E?HP ? H?E?HX ? H?E?H` ?vH?E?Hh ?jH?E?Hp ?^H?E?Hx ?RH?E?H? ?FH?E?H? ?:H?E?H? ?.H?E?H? ?"H?E?H? ?H?E?H? ?
H?E?H? L?}?H?H?5?" L?5?? L??A??H? x" L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??H? 1" H??X???? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A?
L??H??A??H??????H??x???H??????H??????HDž???? HDž???? HDž???? HDž???? H?5J H??????H?D$(?????????D$H??????H?D$?????????$L??A??? ?????? ?? H??? A? A? ? ?? H?? ?"H?? ?H?? ?H?? ?H??( A? ? H?E?A? ? H??0 ?.H??8 ?%H??@ A? A? ?yH??H ?H??P A? A? ?[H??X ?FH??` ?=H??h ?4H?À ?+H?Ð ?"H??p ?H??x ?H?Ø ?H?È A? A? ? H?E?H?H?5? L?-?? L?e?L??A??H? ? L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??H? C H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H?????? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A? L??H??A??H??x???? A?
L??H??A??H??@???? A? L??H??A??H??????? A? L??H??A??H??????? A?
L??H??A??H??????? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H?U?? A? L??H??A??H??h???? A? L??H??A??H??(????? ?? ??H ???? H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H?? ???H??(???H??0???L??8???L??@???HDžH??? H?5Y H??H???H?D$(???8?????D$H??0???H?D$??? ?????$H?}???? ?
????????? 1҃???H?? H? H?5? H?]?H??1?A??H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H??P???H??X???H??`???L??h???L??p???HDžx??? H?5? H??x???H?D$(???h?????D$H??`???H?D$???P?????$H??A???K ?F??? ?G H?
$& Hc?H??????? ? HD?H?5? H?]?H??1?A??H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H??????H??????H??????L??????L??????HDž???? H?5? H??????H?D$(?????????D$H??????H?D$???????????H?E?H? A? A? ? H?E?H? ?.H?E?H? ?"H?E?H? ?H?E?H? ?
H?E?H? A? A? ?? H?E?H? ?=H?E?H? ?1H?E?H? A? A? ? H?E?H? ?
H?E?H? A? A? ?vH?E?H ?^H?E?H ?RH?E?H ?FH?E?H( ?:H?E?H8 ?.H?E?H ?"H?E?H ?H?E?H@ ?
H?E?H0 A? A? H??# ????tD??A??I9??t D?u?H?5? L?%?? L?u?L??H??A??H? ? L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??A? L??H??H??`???L??A??H? Q H??????? A? L??H??A??H?????? A? L??H??A??H??0???? A? L??H??A??H??X???? A? L??H??A??H?U?? A? L??H??A??H??p???? A? L??H??A??H?????? A?
L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??????? A?
L??H??A??H??x???? A? L??H??A??H??@???? A? L??H??A??H?? ???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H??????? A? L??H??A??H??(????? ?? ??H ???? H?E?H?HH??H??HI?H??Hc?????H?? ???HDž??? H?????D???M?H?????H?? ???HDž(??? H?5 H??(???H?D$(????????D$H?????H?D$??? ?????$L????? ?? ?ȃ?????? 1҃???H?? H? H?5? L??1?A??H?E?H?HH??H??HI?H??Hc?????H??0???HDž8??? H??@???D???M?H??H???H??P???HDžX??? H?5J H??X???H?D$(???H?????D$H??@???H?D$???0?????$L???w????A??? ?? H?
? Hc?H?????(????? ? HD?H?5? L??1?A??H?E?H?HH??H??HI?H??Hc?????H??`???HDžh??? H??p???D???M?H??x???H??????HDž???? H?5r H??????H?D$(???x?????D$H??p???H?D$???`????#???L??L???? H?X H??HI?H??L??H??H??L??`???I?I??H?E?H?8??? I9??f H?5? L?u?L??L??1?A??H?E?H?HH??H??HI?H??H??p???H??0???H??????H??????H?? ???HDž??? H?????HDž??? H?5? H?????H?D$(????????D$H?? ???H?D$?????????$L???K L??P???I???? H?]?H??? A? ? H?E??$ ??@????v H?E?H? ?? H?E?H ?? H?E?H ?? H?E?H ?? H?E?H ? H?E?H ? H?E?H( ? H?E?H0 ? H?E?H8 ? H?E?H@ ?vH?E?HH ?jH?E?HP ?^H?E?HX ?RH?E?H` ?FH?E?Hx ?:H?E?H? ?.H?E?Hh ?"H?E?Hp ?H?E?H? ?
H?E?H? H?H?5? L?=?? L?e?L??A??H? ? L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??A? L??H??H??`???L??A??H? 5 H??????? A? L??H??A??H?????? A? L??H??A??H??0???? A? L??H??A??H??X???? A? L??H??A??H??p???? A? L??H??A??H??8???? A? L??H??A??H?????? A?
L??H??A??H??x???? A? L??H??A??H??@???? A? L??H??A??H?? ???? A?
L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H??????? A? L??H??A??H???????? ???Hc?H?5? L??1?A??H?????H?H H??>H??HI?H??H?M?H?Q?H??~H??HI?H??Ic?H??????H??????H??????HDž????? HDž???? HDž???? H?51 H??????H?D$(?????????D$H??????H?D$?????????$L??A??H?Ę
[A\A]A^A_]?H?5? ?? H?]?H??1?A??H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H??????H??????H??????L??????L??????HDž???? H?5l H??????H?D$(?????????D$H??????H?D$??????????????`????h H??`???? ?W H??(????I H?]?H?? Hc?P???H?E?A? A? ?????H?5? ?? L??1?A??H?E?H?HH??H??HI?H??Hc?????H??????HDž???? H??????D???M?H??????H??????HDž???? H?5u
H??????H?D$(?????????D$H??????H?D$????????&???H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H?????H?????H?? ???L??(???L??0???HDž8??? H?5? H??8???H?D$(???(?????D$H?? ???H?D$??????????H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H??????H??????H??????L??????L?? ???HDž??? H?5K H?????H?D$(?????????D$H??????H?D$????????????H?E?H?HH??H??HI?H??H?? ???H??@???H??x???H??@???H??H???H??P???L??X???L??`???HDžh??? H?5? H??h???H?D$(???X?????D$H??P???H?D$???@????X???H?E?H?HH???H?H??H??H?? ???H??@???H??x???H??p???H??x???H??????L??????L??????HDž???? H?5# H??????H?D$(?????????D$H??????H?D$???p????????H??x???H?U?H?H??H??H ?H?? ?$ H?H?}?? H?]?H??? ????H?E?H?HH??H??HI?H??Hc?????H??????HDž???? H?? ???D???M?H?????H?????HDž??? H?5W
H?????H?D$(????????D$H?? ???H?D$????????:???H?E?H?HH??H??HI?H??Hc?????H??????HDž???? H??????D???M?H??????H??????HDž???? H?5? H??????H?D$(?????????D$H??????H?D$???????????H?E?H?HH??H??HI?H??Hc?????H?? ???HDž(??? H??0???D???M?H??8???H??@???HDžH??? H?57 H??H???H?D$(???8?????D$H??0???H?D$??? ???????H?E?H?HH???H?H??H??Hc?????H??P???HDžX??? H??`???D???M?H??h???H??p???HDžx??? H?5? H??x???H?D$(???h?????D$H??`???H?D$???P???????H?E?Hc?????H??????HDž???? H??????D???M?H??????H??????HDž???? H?5, H??????H?D$(?????????D$H??????H?D$???????????H?È ?? H?Ð ?? H?Ø ? H?à ? H?è ? H??? ? H??? ? ??@????b H?]?H??P ?jH??X ?aH??` ?XH??h ?OH??p ?FH??x ?=H?À ?4H?ð ?+H?ø ?"H??? ?H??? ?H??? ?H??? L?}?H?H?5? L?5?? L??A??H? ? L??H??H?U?H?M?E1?A??A? L??H??H??????H??????A??A? L??H??H?U?H?M?A??H? p H??X???? A? L??H??A??H??p???? A? L??H??A??H??8???? A? L??H??A??H?????? A? L??H??A??H??@???? A? L??H??A??H??????? A? L??H??A??H??????? A? L??H??A??H??????? A?
L??H??A??H??P???? A? L??H??A??H??H???? A? L??H??A??H?U?? A?
L??H??A??H??h???? A? L??H??A??H?5X ? L??1?A??H??x???H?H H??>H??HI?H??H?M?H?Q?H??~H??HI?H?? ???H??@???H??H??????H??????H?? ???HDž???? HDž??? HDž??? H?5? H?????H?D$(????????D$H?? ???H?D$???????????1??u?H?? ???H??@???H?U?H??????H??????H??????L??????L??????HDž???? H?5^ H??????H?D$(?????????D$H??????H?D$???????? ???H???
??H?5=? H?V? ? H??1??????H?=? H?P? ?? 1??????H?=? H??H H?
>? ?? 1?????H?=ʗ H??H H?
4? ?? 1?????H?=?? H?H H?
? ?L 1??y???H?=?? H?)? ?k 1??_???H?=n? H?DH H?
?? ?U 1??>???H?=M? H?#H H?
?| ? 1?? ???H?=,? H?H H?
?? ? 1??????H?= ? H??G H?
? ?0 1??????H?=? H??G H?
?? ?1 1?????H?=ɖ H??G H?
? ?2 1?????H?=?? H?~G H?
?? ?l 1??x???H?=?? H?]G H?
? ?m 1??W???H?=f? H?? H?B H?
x? ?? 1??????)??H?5ک H?L? ? D??I??1??????H?=?? H? C ?? 1??????H?=ސ H?C ? 1?????H?=Đ H?e? ?h 1?????H?=?? H??A H?
Ӧ ?? 1??z???H?=?? H?_A H?
R? ?? 1??Y???H?=h? H?>A H?
y? ??
1??8???H?=G? H? A H?
}? ??
1?????H?=&? H?KB ?{
1??????H?= ? H?1B ?? 1??????H?5̧ ? L??1?????H?5Χ ? 1?????H?=ɏ H?ߧ ??
1?????H?=?? H??@ H?
? ?B 1?????H?=?? H?d@ H?
ǧ ?( 1??^???H?=m? H??A ? 1??D???H?=S? H?xA ?O 1??*???H?=9? H?^A ?? 1?????H?= ? H?DA ?? 1??????H?=? H??? ?` 1???????{P?? ??H?5?? H?? ? D??I??1?????H?=Î H??@ ?O 1?????H?5?? ? L??1??d???H?5?? ? 1??Q???H?=?? H??? ??
1??W???H?5?? ? 1??$???H?=S? H??? ?w 1??*???H?=9? H?? H?
?? ?? 1?? ???H?=? H??? ? 1??????H?5?? ? ? 1?????H?=? H??? ?? 1?????H?=̍ H?? ?? 1?????H?=?? H??> H?
?? ?? 1?????H?=?? H?g> H?
? ?? 1??a???H?=p? H?ߢ ?? 1??G???H?=V? H?,> H?
?? ?? 1??&???H?=5? H? > H?
?? ?? 1??????P~???}??????A????}???}???}??????]???????????É??????A???????????????????????????????2???U???????????????B???????????????P~??P~?????????????????????????????B???????C???????????????%??????;???????????????????????3??????????՛??????????Nj?????????x???????????????????????ʈ?????????p???|???????????????????????????????????????????????????????????????????????
?????????????????????????????????????????????????????????????????%???4??????@???L???X???d???p???|???????????????????????????????1??? ???'???6???S???S???E???T???c???S???r???????????????????S??????????????????????????????? ???S???S???S???S???S??????????:???U???a???????????m???y????????????????????????????????????????????????'???3???????K???????????????????????W???|??? ???????,???????????????=???????J???=???????J???B???W???q???z????????????????????????????????????????'???????3???????????????????????????????????????????????????K??? ???x??????????? ??? ??????????????? ????????????????????? ???,???5???>???G???P???Y???b???k??? ??? ??? ??? ??? ???t???????_??????????? ??? ??? ??????? ???&??????? ???&???????????????q???F???6???&???M???????h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h???h????????????y??q???q?????? ??????ky??????????????????????UH??SPH??H??H??? H??t?? t"H??? H??t?: tH??? H??t?> u 1?H??[]?D?CPI??Ew??L?
? Oc?M?A??H??[]???????? ??H??[]Ë???? ??u????s??? ???u??? t??? u??: ?
H?{(?????H??[]Ã:?n????>?e???H? ?W?????? ?H????????H???s H??? H??[]?-????;??H??[]Ã?????????U?????????????????????9???9??????9???9??????D???????????????????????????D??????D???u???u????????????U???U????????????????????????????????D???????????????????????????????????????????????????????????????????????????????????????fff. ? UH??AVSI??H??????H??? H??HD?H?IH?IX?q??~IH?? H??H? 1?????H?=D? H??$ H?
?X ?? 1??????H?=#? H??$ H?
?X ?@ 1??????H?=? H??$ H?
?T ?C 1?????H?=? H??$ H?
?X ?D 1?????? H??H?}??9M H????L ?I??L????L ?@I???TI???OI??M?u I?E M??uM??? L???L L???M M???I???6I??L????L ?I??H????????L H???????H?H??u I?L???PL????L ?M H????L H???$L H????????d???????????????????????????????????????????????????????????????????????????????????????????????????????????????*???????????????????d???d???d???d???UH??AWAVAUATSPI??L??? H??? ?FP??uFH?CH?E?M?gM?oL???!?????t&H????????t?; uI?? |I?? | ?H?}? mA?GP?? ubI??? ?8 uVI??? ?8 uJL?????????t>H?????????t2L????????t&H???????????uH??H??[A\A]A^A_]?l???1?H??[A\A]A^A_]?ff. ? UH??H??]?Ӄ?? UH??H?=$i ? ???UH??SPH??H??? H??H?H??X?J H?{?J H?{?QJ H??H??[]?UJ UH??SPH??H?X? H??H?H??X?_J H?{?hJ H?{?J H???J H??H??[]?J ? UH??SPH???J H??H??[]?J H???|???fff. ? UH??AVSH??H??L??? L;?? }?f?H??? I??H?? H??? H??I??H??H?? ??? I??L;?? |?H???I H??[A^]?? H????I L?u?L???I H??L???XI H?}??+I ??I ??H??H?}??I ??I H????H H??????UH??AVSH??H??H?}??.I H?3H? H?8?K H?CH?{H{?tH?H?D???L?3H? M??tL????H L???%I H??? I 1?H??[A^]?H??H?}?? H???nH ? UH??AVSH? H? H??t+L?3H? M??tL???dH L????H H??[A^]?H [A^]? UH??AVSH?? H?k? H? H?E?H?B H??HI?H?????S H????H??|1???y?O L????????
?T ???O ??W???}%?O ??}-?T ff. ? M??I??E1??6 D ġz????????a ???y ? ģy4^ I??I?I??I??? ??|r???|z??A|B??A|J??A|I`?A|A@??|y ??|1??{r???T???y????_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???zz???T???_???z:??T???_???^???W???.?u???????^??????fff. ? A?? E1? ? E??A????? D?Ӄ?D?s????????H??H?H??????I??I?I??????ġx(? H? H?E?H??H??????H???| I??H??H?F H??HI?H?????_ H?? ??H??? HM?H??????H?RH??????H??H??????1?L??@???H??????H??????ff. ? H??????H??????L?$?L?????? ? ? L??L????w?????H??????I?<? H??R????????H??????I?<? H??d????????I?< ? H??v?????????o H ??@?????R???f??????f????????d???f????????v???f????????n?B?????y"?T?????n?f?????l???~?x?????yX???y???n?F?????}8???~?X?????}X???}? ??n?j?????}Y???}????}X?|?????}????????n?J?????q"?\?????? ?????n?n?????yXU???l???y???n?N?????}8???}X?`?????}? ??n?r?????}Y???}????}XM???}????????? ????????????|H??8???I?G@????????|G I??HI??I???7???H??????H???????Hc?H;??????????H??????H?A H??HI?H?
?? H? H;M?u H??H?H??H?? [A\A]A^A_]??w???w??= ffff. ? UH??AWAVAUATSH?? H??????H?.? H? H?E?H??H??????H??? I??H??H?F H??HI?H?????? H?? ??H??? HM?H??????H?RH??????H??H??????1?L??@???H??????H??????ff. ? H??????H??????L?$?L?????? ? ? L??L????w?????H??????I?<? H??R????????H??????I?<? H??d???????I?< ? H??v?????????(%X ??@?????R???f??????f????????d???f????????v???f?????????B??????T??????f??????x?????m???}???????????W???? ??????J??????\??????n?????]???m???}???????????W???? ???????????|G H??8???I?G@?????????|I??HI??I???????H??????H???????Hc?H;??????h???H??????H?A H??HI?H?
? H? H;M?u H??H?H??H?? [A\A]A^A_]??w???w?K; fffff. ? UH??AWAVAUATSH??X H??????H??? H? H?E?H??H??????H???? I??H?A H??HI?H?????? I??J? ? H????????H??? HM?H??????I??1?L??@??? @ H??????H??????H? ?H?????? ? H??????? H??L????w?f???N?,#? L??H??R????N???M? L??H??d????7???M? L??H??v???? ???M? L??H?u?? ???M? L??H?u??????M? L??H?u??????M? L??H?u????????(%?
??@?????R???f??????f??????H????????d???f????????v???f???????E?f???????E?f???????E?f???????E?f?????????B??????T??????f??????x?????m???}???????????W??????????E???M???U???]???m???}???????????W???????????J??????\??????n?????]???m???}???????????W???? ?????E???M???U???]???m???}???????????W???? ?????|Gp?????????|????????|G`?????????|G@?????????|G I?ǐ H??H???????H???????????Hc?H;??????b???H??????H?A H??HI?H?
i? H? H;M?u H??H?H??H??X [A\A]A^A_]??w???w??7 ? UH??AWAVAUATSL?E?H?MЋU?G ??IǍJ??Iʃ??? ???? ?? ??HcЉ?H?E???H?E?H?E?L?HH?U?H?? H??H?E?H?E?H??H?E?1???o?D ??o
g H?u? H??H??H?M?H?E?H?E?H??H?M?L?$???W?L?m?L?M?E1ҐK??L??H??J? QH?U??
A??A?t?A?|?A?D?L?[? ??z ???a! ???a! ? ??a! ?0??y$?? D A?|
?E?D
A?\
?A?T
A?t
E?|
A?D
E?\
E?t????yn???Q l????Q ,???Q l???q???????y!???????y!???n???yX???A@???yn???yX???A@?????A?|????n???I t????I t???I t???r???q???????y!???????y!??yn??ByX???9@??yn??ByX???9@???????r?A?T????n???A |????A |???????A t???q???????y!???????y!??yn??ByX???9@??Ayn??ByX???9@???????r?A?T????n???A |????A |???A |?Źq??9???By!???????y!??yn??ByX??B1@??Ayn??ByX???1@?Ź????r???????????[???Y???Y??H??H???.???I??I??HI??"L;U??????H?u?H?E???H?E?H??L?M?LM?H;E??c???? ??H?Q??ȃ?H??s1??M???H?~p1?????fff. ? ??G???G???G???G???G???G???G?H????H??H9?u?H??t%H??H?H??1?????f D ??H??H9?u?[A\A]A^A_]? ? UH??AWAVAUATSH??hH?M??UH?A? H? H?EЍG ??IǍJ??Iʃ??? ???? ?$ ??H???H?M?H?E???H?E?1???W???o
?@ ??o? H?u?f D H??H??H?H?M?H?E?H?E???)E?H??H?M?H??H?E???(]?1?ffff. ? H??L?M?M? ?????hX ????hp ????h? ????h? ????h? ????h? ????h? ?~???h ?t???h ?j???h1 ?`???hJ ?V???hX ?L???hg ?B???h? ?8???h? ?.???h? ?$???h? ????h? ????h? ????h
?????h ?????h% ?????h4 ?????hC ?????hR ?????hm ?????h} ????h? ????h? ????h? ????h? ????h? ????h? ?z???h? ?p???h? ?f???h? ?\???h
?R???h ?H???h% ?>???h5 ?4???hD ?*???hS ? ???hb ????hq ? ???h? ????h? ?????h? ?????h? ?????h? ?????h ?????h+ ?????hB ????hO ????h^ ????h} ????h? ????h? ????h? ????h? ?v???h? ?l???h ?b???h3 ?X???hN ?N???hk ?D???h? ?:???h? ?0???h? ?&???h? ? ???h? ????h? ????h? ?????h
?????h
?????h'
?????h6
?????hE
?????hV
?????he
????hu
????h?
????h?
????h?
????h?
????h?
?|???h?
?r???h?
?h???h ?^???h ?T???h& ?J???h7 ?@???hK ?6???ha ?,??? ??? ? ? @ ?*BL?'7= ??#?ٿ @ F @?I@ @@ ?@ ? @K;??? @? r1????? ?B?+= 3?*>??? ? @C ? A ?? ?A ? ?? ( 0 8 ???ff@ ????=o?: 0C 0E 0C 0E ??'7 ??'7??8fff??< ??'7d o?:fff?w??w?+2??'7o?: ???????????? ?? ?? ?? ?? ?> ?> ?> ? 4? ?? ? ? ? ?> ?? ?? ?C % C ? @ ? @ ? + # @ ? T ?? W ?? [ ?? J? _ d h l ?? ?? `? ?? ?? ??
! ? A?? pA ?? ?A ?A ?B???> ????????? ?? @ @ ?;??L=fff?} ?& ?@ ???? B @ ??? = |B? @ ? @ ?? A @ ??? ?A??? ?> > ?? ? ?@ @ w?+2??L>????X9??̼?+ ?? `? ?? ?? `????3ff?? ?B ???|? ???̾??????L???̽???>???> ?? ???????????????? 00000000 00000000 ???????? ???????? 00000000 ????????
????????????????
Q Q Q Q Q Q Q Q 0000000000000000???????????????? ????????
&5EYq ???????????? @ ( 0 ?????????????? ? ??
@ ????????????????
?? ? ?? `? `? ?? `? ?? ? > > ? > ? `? ?? ?? `? ?? `? ? ? ? ????????????????
???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????????????????????????????????????? ???????????????????????????????? ???????????????????????????????? ???????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????
???????????????????????????????? ?????߿?????߿?????߿?????߿???????????????????????????????? @ ? @ ?
@ ? @ ?
?????߿?????߿?????߿?????߿
???????????????????????????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
???????????????????????????????? ?? ? ?? ? ?? ?? ? ?? ? ??
?
?
| | | | | | | | | | | | | | | | Q? +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ?????
? ????????? ??!"?$??'(??+?-.?0??3?56??9:? $ >,> , $ $, > $ > ,>,,> > , > $> $$$$>,$ >$,>$ ,>, ,,,, 4$44 >$ >4 > $> 4> $ >$ , > 4$ 44 $ ,$ ,$ , $, 4$, >, ,4 > $> 4 $ , 4 >4 > > > $,, $ > $> , $ ,$ ,>, $,$>, > >4 >,$> , ,4 > $$ 4$ , , > $ >$$ >$ , 4, , ,,, $ 4 4 4 44 > 4> $$> $, $> $ ,$>,$$> $$> $$$4$44$> $,$ $$$$ ,$$$4$$,,$ $,$>,$,>$ >$ >$ >$ , $,>, ,4 ,4 ,,, ,$ , ,>, , , , $, $,>$,>$,,, ,,,4,$>,$>,$4$$44$4$44 4 4 4> 4$44 44 4$$$4,,4,,4 44 >4 >4 >,>>> > >,>4 >$ > >,$>,> >, > >4 > $> $$>,>,>$,> 4>
??????????????? ??????????????????? ? ????? ????????????????????????????? ? ???? ? ????? ???? ???? ???????????????????????????? ????????????????????????? ?? ??? ?? ??? ? ? ???? ? ??? ? ??? ? ??? ? ??? ? ??? ?? ??? ? ??? ? ???? ??? ??? ??? ? ???? ??? ???? ??? ? ??? ? ???? ??? ??? ??????????????????????????? ???????????????????????? ? ??? ? ??? ???? ??? ??????????????????????? ? ??? ??? ??????????????????? ? ?? ??? ?? ?? ?? ?? ?? ?? ?? ? ?? ? ? ?? ? ?? ? ??? ? ?? ? ?? ? ? ?? ? ?? ? ?? ??? ??? ?? ?? ?? ?? ?? ?? ? ??? ? ?? ? ?? ? ????? ?? ? ??? ???? ?? ? ??? ?? ?? ?? ???? ??? ?? ??? ?? ?? ?? ?? ? ?? ??? ?? ?? ? ??? ?? ?? ?? ??? ? ?? ? ?? ?? ??? ? ???? ??? ?? ??? ???? ?? ? ?? ?? ?? ? ?? ? ??? ?? ?????????????????????????? ???????????????????????? ? ??? ? ??? ??? ??????????????????????? ??????????????????? ?? ?? ? ? ??? ? ?? ? ??? ? ?? ? ?? ?? ???? ?? ? ?? ?? ? ??? ?? ?? ? ??? ?? ?? ?? ?????????????????????? ??????????????????? ? ?? ? ?? ?? ? ?? ??????????? ?????????? ???????????????? ??? ? ? ??? ?? ??? ? ??? ?? ??? ? ??? ? ?? ?? ?? ? ?? ??? ?? ? ?? ?? ?? ? ? ?? ?? ?? ? ?? ? ?? ? ? ?? ?? ?? ? ?? ? ??? ? ?? ? ??? ? ?? ?? ? ? ?? ? ? ?? ? ??? ? ? ?? ? ? ? ? ? ?? ? ? ? ?? ? ?? ? ? ? ? ? ? ? ? ?? ? ??? ? ? ? ? ?? ? ?? ? ? ? ? ? ? ? ??? ? ? ? ?? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ?? ?? ?? ? ?? ? ?? ???? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ???? ? ??? ? ??? ?? ?? ? ?? ?? ?? ? ?? ? ??? ? ?? ? ?? ? ? ? ? ? ? ? ??? ? ?? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ?? ?? ? ?? ??? ? ?? ? ? ? ?? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ???? ? ?? ??? ? ? ? ?? ? ? ? ? ? ? ? ???? ??? ? ? ??? ?? ? ?? ?? ? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ? ?? ? ? ? ???? ??? ?? ? ? ? ??? ?? ??? ? ? ?? ? ? ? ? ??? ?? ? ? ??? ?? ?? ? ?? ? ?? ? ? ? ? ?? ? ? ? ? ?? ? ? ???? ??? ?? ? ? ? ??? ?? ? ? ? ? ?? ? ? ? ?? ? ?? ? ? ? ? ?? ?? ? ?? ? ? ? ?? ? ?? ? ?? ? ?? ? ?? ? ? ? ? ? ? ? ? ??? ? ? ? ?? ? ? ? ??? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ??? ? ?? ? ? ? ? ? ? ?? ? ? ??? ?? ? ? ? ??? ? ? ?? ? ? ? ? ??? ? ? ? ? ? ? ?? ? ?? ? ? ? ? ? ?? ? ? ?? ? ? ? ?? ? ? ? ? ? ??? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ????????????????????????? ??????????????? ????????????? ? ??? ? ??? ??? ? ??? ??????????????????????? ???????????? ?????????? ?? ?? ? ? ??? ? ?? ? ?? ? ???? ??? ? ?? ? ?? ??? ?? ?? ? ??? ?? ?? ?? ? ???? ?? ?? ?? ?? ?????????????? ? ??? ??????????????????? ? ?? ? ?? ?? ?? ?? ??????????? ? ?? ?????????????? ??? ?? ?? ? ?? ? ?? ? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ?? ? ? ? ?? ? ? ? ?? ? ? ???? ? ? ??? ? ? ?? ? ? ?? ? ?? ? ? ??? ?? ? ?? ? ? ? ?? ? ? ? ?? ??? ? ?? ? ? ? ? ? ? ?? ? ? ? ??? ? ? ? ?? ? ? ? ??? ? ????????????????????? ??????????????????? ? ?? ? ??? ?? ?????????????????????????????? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ??? ? ? ?? ? ????????????????????????????? ? ? ? ? ? ? ????????????? ????????? ????? ? ???? ???? ???? ? ??? ? ??? ??? ? ??? ??? ? ??? ? ??? ??? ? ??? ??? ???? ??? ???? ?? ??? ?? ? ?? ?? ?? ?? ?? ?? ? ? ?? ? ?? ? ?? ? ? ?? ? ?? ?? ?? ? ?? ? ?? ? ?? ?? ?? ? ?? ? ?? ? ?? ?? ?? ?? ?? ?? ? ?? ?? ?? ?? ? ? ?? ? ?? ?? ?? ? ?? ?? ?? ?? ? ?? ? ?? ???? ? ??? ? ??? ??? ??? ?? ?? ? ?? ?? ?? ? ?? ?? ??? ? ?? ?? ???? ? ?? ? ?? ? ?? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ?? ? ?? ? ?? ? ? ? ? ? ? ? ??? ? ?? ? ? ? ??? ? ??? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ??? ? ?? ? ? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ???? ? ??? ??? ??? ??? ?? ?? ? ?? ?? ?? ?? ? ?? ? ?? ? ?? ?? ?? ??? ? ?? ? ?? ?? ?? ?? ??? ? ?? ? ?? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ????? ? ???? ???? ???? ???? ? ??? ??? ??? ???? ? ??? ??? ??? ?? ? ?? ?? ?? ? ?? ? ?? ?? ?? ?? ?? ? ?? ??? ??? ?? ? ?? ???? ??? ?? ?? ?? ? ??? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ???? ??? ? ?? ?? ?? ??? ? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ? ? ? ? ??? ??? ? ?? ? ?? ?? ?? ? ?? ?? ??? ?? ??? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ???? ??? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ??? ??? ? ?? ?? ?? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ??? ?? ?? ? ? ? ? ?? ? ?? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ?? ? ? ? ? ? ???? ??? ??? ? ?? ?? ??? ?? ? ? ? ? ? ? ? ? ??? ?? ?? ? ? ? ? ?? ? ?? ? ? ?? ? ? ?? ? ? ? ? ? ? ?? ? ? ?? ? ? ? ? ? ? ??? ??? ??? ??? ?? ?? ?? ? ?? ??? ?? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ??? ?? ?? ? ? ? ? ? ? ?? ? ??? ??? ?? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ??? ? ? ??? ? ? ? ? ?? ? ? ? ? ? ?? ? ? ? ? ???? ? ??? ??? ??? ??? ? ? ?? ?? ?? ?? ?? ? ?? ?? ? ?? ? ?? ?? ?? ???? ?? ?? ??? ? ? ?? ? ?? ? ?? ? ? ? ? ? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ?? ??? ? ?? ? ? ? ? ? ? ? ?? ? ?? ? ? ? ? ? ?? ? ?? ?? ?? ?? ??? ?? ?? ??? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ?? ?? ?? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ??? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ??? ?? ? ??? ? ?? ? ??? ? ?? ? ? ? ? ?? ? ??? ?? ? ? ? ? ?? ? ?? ?? ?? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ??? ? ?? ?? ? ? ?? ? ? ? ? ??? ?? ? ? ?? ? ? ?? ?? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ???????????????????????????????????????????? ? ????? ??? ? ???? ??? ??? ??? ??????????????????????? ??????????????????? ?? ?? ? ? ??? ? ?? ? ?? ? ?? ? ?? ?? ??? ? ?? ? ???? ??? ?? ?? ??? ?? ??? ?? ?? ?? ? ?? ? ??? ?? ?? ?? ?????????????????????? ??????????????????? ? ??? ?? ?? ?????????????????? ??????????????? ?? ? ?? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ??? ? ?? ?? ? ? ? ? ? ? ? ? ? ???? ? ?? ? ? ??? ? ? ?? ? ? ? ? ? ? ?? ? ? ?? ?? ? ? ? ? ? ??? ?? ? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ? ? ? ?? ??? ? ? ? ? ? ? ? ? ? ? ? ? ?? ????????????????????? ??????????????????? ?? ?? ? ?? ? ??? ?? ?? ?? ?????????????????? ?????????????? ?? ? ?? ? ? ? ?? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ??? ?? ? ? ? ? ?? ? ? ? ? ? ????????????????? ?????????????? ? ? ? ?? ? ????????????? ????????? ???? ? ??? ??? ? ??? ??? ??? ?? ?? ? ?? ? ?? ?? ? ?? ?? ?? ?? ??? ? ?? ? ?? ?? ?? ?? ??? ? ? ?? ? ?? ? ?? ? ?? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ??? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ?? ? ?? ?? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ? ???? ? ??? ??? ?? ?? ?? ? ?? ?? ??? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ? ? ?? ? ???? ?? ??? ?? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ?? ? ? ? ?? ? ? ??? ? ?? ?? ? ? ? ??? ? ? ? ?? ? ? ? ? ?? ? ? ? ? ? ? ? ??? ?? ?? ?? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ?? ?? ?? ? ? ? ? ? ? ? ??? ?? ? ? ? ??? ?? ? ?? ? ? ? ? ?? ? ? ?? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ???????????????????? ??????????????????? ? ?? ? ??? ?? ?? ?????????????????? ?????????????? ?? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ??? ?? ? ? ?????????????? ?? ?????????? ? ? ? ? ? ? ????????????? ????????? ?? ?? ?? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ?? ?? ? ? ? ? ? ? ? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ?? ? ? ???????????????????????????? ? ?? ? ? ????????????? ????????? ?? ? ? ? ?? ? ?? ?? ? ?? ???????????????????? ? ? ???????? ????????????
&5EYq????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
" ( * A D P X a d ? ? ? ? @?? "?!$@BH`???? Fi?? @?? (ADPR? @
!@`??? PZ?$E %I?b @? ?Q
D a ? ? )!H! ""$$$@$V$ %A%d%?&( (?(D*@@@@!@$@@@H@V@`@?@?@?@ A AaA?A?ABBHBVBhB DD D?D?DE$E FHHH@HEH IXIaI?IEJ?J PPPP P?P?PQBQ?Q?R?T?T
UVTV XXXdX@YZ``@`h` aUab`b ddee?eBh ??
?A?????@??????@?`? ?F??? ?@?`??????$?@??????E? ?"?D?Q??? ??P??? ? ??P??
" % ( A D F I P R U X a d ? ? ? ? ? ? ? ? !$@BEHQT`h??? ADPU?? !$@BEHQTV`??? ADPa?&@B?
%ADPX??? @ ? ?
(
?
?
!$@BEHQT`j??? ADP???@` ADIPd?@ I@T? fQ?
A D P ? ? !!!@!H!e! """?"?"$$$)$@$ %A%R%?%&&?& ((
( (U(?(?(h)?)* *?*?*?*@@@ @@@@@!@$@@@B@E@H@J@Q@T@`@e@?@?@?@ AAAAAA AAADAPA?A?A?ABBBB)B@B DDDDDDD DADDDPD?D?DEEE$E@E?E F
FDFPFHHH@HEHTHbH IIDIPIiIJ PPPPPP P(PAPDPPP?PQQQQ@QBQ RDR?RTTT!T@T`T?T?T UU?U?U!VhV?V XXAXPX?XY@YBY?Z```@`T`b`?`?` a$bJb?b dde@eEe?ehjh%ijTjbj ?????? ?*?A?D?P?????????????@?Q?Y? ? ??????????????@?`??? ?D????j? ??%?Z????????"????????????@?V??? ?"?d?V??? ??D?P?X?)?????A?Q???I??`? ???
? ?*???Q?Y??? ???*?????@???e???
? ?"?(??????????(?*?????
" ( * E Q T V e ? ? ? ? ? ? ? ? ? %AFIRUZadfh?????
"(*EQYdi???????????%AIUZde???)@EJPQTUVY`behj??????????ADPRUX`afi????
"(*EQVe????????? $ % A P Q U a d i ? ? ? ? ?
"
(
*
E
Q
Y
a
e
?
?
?
?
?
?
?
?
?
?
$%ADPUXadei????? !$)EJPQRTUVY`e?????%@FIRUXZdf????? !&AEFHJQTUVYbeh????????????
"%(*ADEFQRTUVYZadefi????????????????!&@BDEHJQUVXYadehij??????%ADFIPUXZ`adfi????!%BDEHQTUVYZ`ej???????? $&DFIPRUXafi????
" % ( * E Q Y a e ? ? ? ? ? ? ? ? ? ? !!!!%!B!D!I!U!X!Z!a!d!e!f!?!?!?!?!?!""
""" """("*"E"Q"V"Y"e"?"?"?"?"?"?"?"?"?"$$$$%$D$E$F$I$R$U$X$Z$f$?$?$?$?$?$?$ %%!%)%@%E%H%Q%T%U%Y%b%e%h%?%?%?%?%?%?%?%?%?%?%&&&&%&A&I&U&`&a&i&?&?&?&?& (((
(( ("(((*(E(Q(T(e(?(?(?(?(?(?(?(?( ))))%)F)I)R)U)a)d)f)i)?)?)?)?)?)?) ***
* *"*(***E*Q*V*Y*e*?*?*?*?*?*?*?*?*?*@@@%@I@R@U@X@Z@a@d@f@?@?@?@?@ AAAA AAAAAA!A&A)AEAHAJAQATAUAVAYAZAeAhAjA?A?A?A?A?A?A?A?A?ABBBB%BABRBUBZBdBiB?B?B?BDDD)DEDHDJDQDTDUDVDaDbDeDhDjD?D?D?D?D?D?D?D?D?DEEE
EEEEEE E%E*EAEDEEEFEIEPEQETEUEVEXEYEaEdEeEfEiE?E?E?E?E?E?E?E?E?E?E?E?E?EFF FFFFF!F$F)F@FBFEFHFPFQFRFUFVFYFbFeFhF?F?F?F?F?F?F?F?FHHHH%HBHIHPHUHXHaHdHfHiH?H?H?H?H?H?HIII
IIIII!I$I&I@IEIJIQIRITIUIVIYI`IbIeIfIjI?I?I?I?I?I?I?I?I?I?IJDJFJIJUJXJZJdJiJ?J?JPPPP PPPP!P$P)P@PEPHPQPTPUPVPYPePhP?P?P?P?P?P?P?P?PQQ Q
QQQQQQQ Q%Q&Q(Q*QAQDQEQFQIQPQQQRQTQUQVQXQYQZQaQdQeQfQiQ?Q?Q?Q?Q?Q?Q?Q?Q?Q?QRRRRR!R$RBRERJRQRTRURVRYRbReR?R?R?R?R?R?R?RTTTTTTTT!T%T(T*TATDTETFTITJTPTQTTTUTVTXTYTZTaTbTdTeTfTiT?T?T?T?T?T?T?T?T?T?T?T?TUUUUU UUUUUUUUU!U$U%U&U)U@UAUBUDUEUFUHUIUPUQURUTUUUVUXUYUZU`UaUdUeUfUhUiUjU?U?U?U?U?U?U?U?U?U?U?U?U?U?U?U?U?U VVVVVV VVVVVV V!V"V$V%V&V(V)VAVEVFVHVIVJVPVQVRVTVUVVVXVYVZVaVdVeViV?V?V?V?V?V?V?V?V?V?V?V?V?V?VXXX XXXX!X*XEXHXJXQXTXUXVXXXYX`XbXdXeX?X?X?X?X?X?X?X?XYYY
YYYYYY%YAYDYEYFYIYPYQYRYTYUYVYXYYYZYaYdYeYfYiY?Y?Y?Y?Y?Y?Y?Y?Y?Y?YZZZZ Z%Z&Z)ZEZHZIZQZUZVZXZYZbZeZhZjZ?Z?Z?Z?Z?Z?Z?Z?Z````%`D`P`U`V`X`Z`a`d`f`i`?`?`?`aaa aaa!a"a&a)aEaIaQaUaVaYaeafaja?a?a?a?a?a?a?abbb@bAbFbUbVbXb`b?b?b?b?bddddd!d&d)d@dBdEdHdJdQdTdUdVdYdZd`dbded?d?d?d?d?d?d?d?d?d?d?d?d?dee
eeeeeDeEeFeIePeQeTeUeVeYeaedeeefeie?e?e?e?e?e?e?e?e?e?e?e?ef ff f&f(f)f@fEfHfJfQfTfUfVfXfZf`fefhf?f?f?f?f?f?f?f?f?f?f?f?fhh%hAhRhUhZhahih?h?h?h?hiiii!i$i&i)i@iAiEiFiHiQiTiUiViYi`ieiji?i?i?i?i?i?i?i?ijjjAjDjIjPjUjXjZjdjejij?j?j?j?j?j ???
? ?"?(?*?E?P?Q?T?V?Y?e????????????????????????%?A?D?I?P?R?U?V?X?Y?d?f?i????????????? ???
?? ?"?(?*?Q?T?Y?e?????????????????????A?D?Q?U?Z?a?d?i?????? ????&?)?@?A?E?H?Q?T?U?V?Y?Z?e?f?h?j??????????????????????%?A?D?I?J?P?U?Y?Z?a?f?j????????? ???
?? ?"?(?*?A?E?Q?T?Y?e?i????????????????????????%?A?D?F?I?P?R?U?Z?a?d????????? ???
?? ?"?(?*?E?Q?T?V????????????????????????%?A?F?I?U?X?Z?i?j???????????????? ?????!?$?&?)?@?E?P?Q?T?U?V?Y?b?e???????????????????????%?D?F?I?P?R?U?X?f?i???????????????&?@?J?Q?T?U?V?X?Y?`?a?b?e????????????????? ???
???????!?%?)?*?A?D?E?F?I?P?Q?R?T?U?V?X?Y?Z?a?d?e?f?i???????????????????????????????????? ?&?)?E?H?I?Q?R?U?V?Y?e?h????????????????????????%?A?F?P?R?U?V?Z?d?e????????????? ?????? ?!?$?&?@?B?E?H?J?Q?T?U?V?Y?b?e?f?j???????????????????%?D?F?I?P?U?X?a??????????? ???
?? ?"?(?*?E?Q?T?V?Y???????????????????? ??????F?I?Q?U?X?Z?a?d?????????????
???"?(?*?E?Q?V?Y?e????????????????????%?A?D?P?T?U?X?Z?a?e?f?h?i???? ?????&?)?B?E?Q?T?U?V?Y?e?j?????????????????????!?%?D?F?J?R?U?V?X?`?b????????????????? ???
? ?"?(?*?Q?T?V?Y??????????????????????!?%?A?P?U?Z?a?f?i????? ???
? ?"?(?*?Q?T?V???????????????????
" % ( A D F I P R U X a d f i ? ? ? ? ? ? ? ? ? !$@BEHQTVY`eh??????? *ADFIPU????? !$&)@BEHJQTVY`be?????????
%(ADFIPRUXad??????? @EHQT`??? %*ADFIPRUXad???? ! @ E H Q T ` ? ?
"
(
*
P
?
!$&@BEHQTVY`beh????????
"%(ADFIPRUXad?????? !$@EQT??? %(ADFIPRUXad??????? !$@BEHQT`??? ADP?? !@BEHQT`?? ADPi?@V % * A D P R U d ? ? ? ? !!!!!!!@!B!E!Q!T!`!?!?!?! "
"""("*"D"P"?"?"?"$$$ $$$$!$$$@$B$E$H$Q$T$`$?$?$?$ %%%%% %A%D%P%f%?%&&&@&Y& ((((A(D(P(?(?()))?)
*"*d*?*?*@@@ @@@@@@!@$@&@@@B@E@H@J@Q@T@V@Y@`@b@e@?@?@?@?@?@?@?@ AAAAAAAA A"A%AAADAFAIAPARAUAXAaAdA?A?A?A?A?A?A?ABBBBBB$B@BEBHBQBTB`B?B?B DDDD
DDDDD D"D%D(DADDDFDIDPDRDUDXDaDdD?D?D?D?D?D?D?DEEE EEEEE!E$E@EBEEEHEQETE`EjE?E?E?E FFFFFF FAFDFPF?F?FHH HHHHH!H$H@HBHEHHHQHTH`H?H?H IIIIII IAIDIPI?I?IJJJ@J PPPPPPPP P"P%P(PAPDPFPIPPPRPUPXPaPdP?P?P?P?P?P?PQQQ QQQQQ!Q$Q@QBQEQHQQQTQ`Q?Q?Q?Q RRRRR RARDRPRiR?RTTT TTTTT!T$T@TBTETHTQTTT`T?T?T?T UUUUUU UAUDUPU?UVVV&V@V XXXXXX XAXDXPXZX?XYYY@Y ZZ?Z?Z```````!`$`@`E`H`Q`T```?`?` aaaaaa aAaDaPa?a?abb@bVb?b ddddd dAdDdPd?deee@eJehe?e f?fhhheh?h i*iBj?j ??????? ?%?A?D?P?R?U?X?a??????????? ?????!?$?@?B?E?H?Q?T????????? ??
???A?D?P???? ?????!?@?B?E?H?Q?T?`??????? ?????? ?A?D?P????????)?@? ????A?D?P?????@?e?"?X?Z??????? ?????$?@?B?E?H?Q?T?`??????? ????A?D?P?Z????@??? ?????? ?A?D?P????????@????? ?F?d????&?@??? ?I?R??? ??
??"?*?A?D?P?????@?e??
?"?(?*????????????@????? ??Q?
?(???T????
? ?"?(????? " ; = A C H J Q U X Z a l x ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? $/;=AGZj???????
'(@BIPR?????????????(PTfy??? "ACEHJQXswx?????????????? +CWa|????
(5:@BPYdf????????':@FRm????? +/|??????? ) 4 U c x ? ? ? ?
!
8
@
F
V
m
?
?
?
?
?
: B Y ? ? ? $ & 4 Q q ? ? ? ? $
E
j
?
?
?
*V`???????!@BT?
" % ' ) + 0 2 9 < ? @ A B D H I M P S W Y ] d q u z ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #)8BDPRV[aegv{???????????????
,.1:@ACFHLQX[ahjx~????????????????????????
"&(,AHKQXZi??????????????
!#%(*78;=@BHKORUY\bi??????????????????????? !:=DIR_`kx????????????
"$')39ACELQS`pz~?????????????????? *04AHJUZdn{???????
!(8BGIKRX]cim????????????????? 9 E G Q r ? ? ? ? ? ? ? ?
$
'
*
6
<
A
C
E
J
Q
Z
z
?
?
?
?
?
?
?
?
?
?
?
?
?
?
& J V i k ? ? ? ?
0 @ P W t ? ? ? ? ? ? ? ? ? ?
#
2
@
C
\
p
?
?
?
?
,2BITce??????????? (Rb???
`? > ?? ?? ? `? ? >???????? 00000000???? @? P? ??.ANSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEE ???????????????????????????????????????????????????????????????????????????????????????????????? ???????? ???????? ???????? ???????? ????
???????? %s:%d: GGML status: error (failed to allocate memory) GGML status: error (operation failed) GGML status: success GGML status: warning (operation aborted) GGML status: unknown /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml.c GGML_ASSERT(%s) failed type < GGML_TYPE_COUNT ggml_numa_init: NUMA already initialized
- ggml_object: type = %d, offset = %zu, size = %zu, next = %p
%s: objects in context %p:
ggml_print_objects %s: --- end ---
NONE wtype != GGML_TYPE_COUNT ctx->mem_buffer != NULL ((uintptr_t) (ctx->mem_buffer))%GGML_MEM_ALIGN == 0 tensor->data fatal error tensor->nb[0] == sizeof(int8_t) tensor->nb[0] == sizeof(int16_t) tensor->nb[0] == sizeof(int32_t) tensor->nb[0] == sizeof(ggml_fp16_t) tensor->nb[0] == sizeof(ggml_bf16_t) tensor->nb[0] == sizeof(float) tensor->op == GGML_OP_UNARY %s (view) ggml_is_matrix(a) ggml_are_same_shape(a, b) ggml_can_repeat(a, b) ggml_can_repeat(b, a) dim >= 0 && dim < GGML_MAX_DIMS a->ne[d] == b->ne[d] ggml_can_mul_mat(a, b) !ggml_is_transposed(a) a->op == GGML_OP_MUL_MAT !ggml_is_transposed(as) ids->type == GGML_TYPE_I32 as->ne[3] == 1 b->ne[3] == 1 ids->ne[2] == 1 && ids->ne[3] == 1 ids->ne[1] == b->ne[2] as->ne[0] == b->ne[0] ids->ne[0] % b->ne[1] == 0 ggml_can_out_prod(a, b) %s (copy) ggml_nelements(a) == (ne0*ne1*ne2*ne3) %s (cont) ggml_is_contiguous(a) ggml_nelements(a) == ggml_nelements(b) %s (reshaped) ggml_nelements(a) == ne0 ggml_nelements(a) == ne0*ne1 ggml_nelements(a) == ne0*ne1*ne2 ggml_nelements(a) == ne0*ne1*ne2*ne3 axis0 >= 0 && axis0 < GGML_MAX_DIMS axis1 >= 0 && axis1 < GGML_MAX_DIMS axis2 >= 0 && axis2 < GGML_MAX_DIMS axis3 >= 0 && axis3 < GGML_MAX_DIMS axis0 != axis1 axis0 != axis2 axis0 != axis3 axis1 != axis2 axis1 != axis3 axis2 != axis3 %s (permuted) %s (transposed) a->ne[2] == b->ne[1] b->type == GGML_TYPE_I32 ggml_is_matrix(a) && ggml_is_vector(b) && b->type == GGML_TYPE_I32 ggml_is_matrix(c) && (a->ne[0] == c->ne[0]) a->ne[1] == 1 ggml_is_vector(b) a->ne[2] == b->ne[0] ggml_is_matrix(b) a->ne[3] == 1 p0 == 0 d0 == 1 a->ne[2] == b->ne[2] a->ne[1] == b->ne[1] (!is_2D || OH > 0) && "b too small compared to a" (OW > 0) && "b too small compared to a" a->ne[3] == b->ne[2] stop > start a->ne[0] >= k ggml_can_mul_mat(k, q) ggml_is_contiguous(mask) mask->ne[2] == 1 mask->ne[3] == 1 mask->ne[1] >= GGML_PAD(q->ne[1], GGML_KQ_MASK_PAD) && "the Flash-Attention kernel requires the mask to be padded to GGML_KQ_MASK_PAD and at least n_queries big" mask a->op == GGML_OP_FLASH_ATTN_EXT TODO: adapt to ggml_flash_attn_ext() changes ggml_is_3d(sx) ggml_is_matrix(c) sx->ne[1] == d_inner n_t >= 0 ggml_is_contiguous(s) ggml_is_contiguous(x) ggml_is_contiguous(dt) ggml_is_contiguous(A) ggml_is_matrix(A) ggml_is_3d(B) ggml_is_3d(s) B->nb[0] == ggml_type_size(B->type) C->nb[0] == ggml_type_size(C->type) ggml_are_same_shape(x, dt) ggml_are_same_shape(B, C) s->ne[2] == n_seqs x->ne[0] == d_inner A->ne[0] == d_state A->ne[1] == d_inner B->ne[0] == d_state B->ne[1] == n_seq_tokens B->ne[2] == n_seqs a->type == GGML_TYPE_F32 qh == kh 2*MAX(qh, kh) - 1 == a->ne[1] ggml_is_contiguous(k) ggml_is_contiguous(v) ggml_is_contiguous(r) ggml_is_contiguous(tf) ggml_is_contiguous(td) ggml_is_contiguous(state) k->ne[1] == 1 v->ne[0] == 1 && v->ne[1] == S && v->ne[2] == H && v->ne[3] == n_tokens r->ne[0] == 1 && r->ne[1] == S && r->ne[2] == H && r->ne[3] == n_tokens td->ne[0] == 1 && td->ne[1] == S && td->ne[2] == H && td->ne[3] == n_tokens ggml_nelements(state) == S * S * H * n_seqs ggml_is_scalar(c) a->flags & GGML_TENSOR_FLAG_PARAM ggml_are_same_shape(a, grad) alpha > 0.0f beta1 >= 0.0f && beta1 <= 1.0f beta2 >= 0.0f && beta2 <= 1.0f eps >= 0.0f wd >= 0.0f && wd <= 1.0f k != GGML_HASHSET_FULL replacements->set.keys[k] == NULL gf->n_nodes > 0 gf->grads node->src[j]->type == GGML_TYPE_F32 || node->src[j]->type == GGML_TYPE_F16 !node->view_src || node->op == GGML_OP_CPY || node->op == GGML_OP_VIEW || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_TRANSPOSE insert_result != GGML_HASHSET_ALREADY_EXISTS dst->size >= src->n_leafs dst->size >= src->n_nodes dst->visited_hash_set.size >= src->visited_hash_set.size dst->grads != NULL cgraph->grads != NULL node->grad->buffer node->type == GGML_TYPE_F32 ggml_is_scalar(node) cgraph->n_nodes + i >= 0 i < cgraph->n_nodes cgraph->size > cgraph->n_nodes rc == GGML_EXIT_SUCCESS || rc == GGML_EXIT_ABORTED node->src[0]->ne[3] == 1 node->src[1]->ne[2] == 1 node->src[1]->ne[3] == 1 cplan cplan->n_threads > 0 cplan->work_size == 0 || cplan->work_data != NULL cplan requested more threads (%d) than available (%d)
%-16s %8x
magic %-16s %8d
version leafs nodes %-16s %llu
eval %-6s %-12s %8s %8s %8s %8s %8s %16s %16s %16s %16s %16s %16s
TYPE OP NDIMS NE0 NE1 NE2 NE3 NB0 NB1 NB2 NB3 DATA NAME cgraph->leafs[i]->op == GGML_OP_NONE cgraph->leafs[i]->src[0] == NULL cgraph->leafs[i]->src[1] == NULL %-6s %-6s %-12s %8s %8s %8s %8s %8s %16s %16s %16s %16s %8s %16s %16s
ARG NTASKS DST SRC wb %s: failed to open %s: %s
ggml_graph_export %s: failed to find tensor, arg = %d, node = %d
rb ggml_graph_import %s: failed to create ggml context
%s: failed to read %s
%s: invalid magic number, got %08x
%s: invalid version number
%s: loaded leaf %u: '%16s', %9zu bytes
%s: loaded node %u: '%16s', %9zu bytes
=== GRAPH ===
n_nodes = %d
- %3d: [ %5lld, %5lld, %5lld] %16s %s
x g n_leafs = %d
- %3d: [ %5lld, %5lld] %8s %16s
========================================
w fp digraph G {
newrank = true;
rankdir = TB;
yellow green lightblue white "%p" [ style = filled; fillcolor = %s; shape = record; label=" %s (%s)| (%s)| %d [%lld, %lld] | %s %d [%lld, %lld, %lld] | %s | %s"; ]
"; ]
pink "%p" [ style = filled; fillcolor = %s; shape = record; label=" CONST %d [%lld, %lld] | ( %d %.1e , src %d }
%s: dot -Tpng %s -o %s.png && open %s.png
ggml_graph_dump_dot f->grad && "ggml_set_param must be called for at least one ancestor" opt-forward.dot opt-backward.dot ggml_is_scalar(tensor) tensor->type == GGML_TYPE_F32 imatrix != NULL start % type_traits[type].blck_size == 0 start % n_per_row == 0 result == nrows * row_size %s: failed to open '%s': '%s'
gguf_init_from_file %s: invalid magic characters '%c%c%c%c'
%s: GGUFv1 is no longer supported. please use a more up-to-date version
%s: failed to read header
%s: array size is too large (%llu)
invalid type %s: failed to read key-value pairs
%s: duplicated tensor name %s
%s: failed to read tensor info
general.alignment %s: tensor '%s' of type %d (%s) number of elements (%lld) is not a multiple of block size (%lld)
%s: failed to initialize context
%s: failed to read tensor data
%s: failed to read the tensor data
key_id >= 0 && key_id < gguf_get_n_kv(ctx) ctx->kv[key_id].type == GGUF_TYPE_ARRAY ctx->kv[key_id].type == GGUF_TYPE_UINT8 ctx->kv[key_id].type == GGUF_TYPE_INT8 ctx->kv[key_id].type == GGUF_TYPE_UINT16 ctx->kv[key_id].type == GGUF_TYPE_INT16 ctx->kv[key_id].type == GGUF_TYPE_UINT32 ctx->kv[key_id].type == GGUF_TYPE_INT32 ctx->kv[key_id].type == GGUF_TYPE_FLOAT32 ctx->kv[key_id].type == GGUF_TYPE_UINT64 ctx->kv[key_id].type == GGUF_TYPE_INT64 ctx->kv[key_id].type == GGUF_TYPE_FLOAT64 ctx->kv[key_id].type == GGUF_TYPE_BOOL ctx->kv[key_id].type == GGUF_TYPE_STRING ctx->kv[key_id].type != GGUF_TYPE_ARRAY ctx->kv[key_id].type != GGUF_TYPE_STRING nested arrays not supported tensor duplicated tensor name tensor not found failed to open file for writing f32 f16 q4_0 q4_1 DEPRECATED q5_0 q5_1 q8_0 q8_1 q2_K q3_K q4_K q5_K q6_K q8_K iq2_xxs iq2_xs iq3_xxs iq1_s iq4_nl iq3_s iq2_s iq4_xs i8 i16 i32 i64 f64 iq1_m bf16 q4_0_4x4 q4_0_4x8 q4_0_8x8 tq1_0 tq2_0 DUP ADD ADD1 ACC SUB MUL DIV SQR SQRT LOG SIN COS SUM SUM_ROWS MEAN ARGMAX COUNT_EQUAL REPEAT REPEAT_BACK CONCAT SILU_BACK NORM RMS_NORM RMS_NORM_BACK GROUP_NORM MUL_MAT MUL_MAT_ID OUT_PROD SCALE SET CPY CONT RESHAPE VIEW PERMUTE TRANSPOSE GET_ROWS GET_ROWS_BACK DIAG DIAG_MASK_INF DIAG_MASK_ZERO SOFT_MAX SOFT_MAX_BACK ROPE ROPE_BACK CLAMP CONV_TRANSPOSE_1D IM2COL IM2COL_BACK CONV_TRANSPOSE_2D POOL_1D POOL_2D POOL_2D_BACK UPSCALE PAD ARANGE TIMESTEP_EMBEDDING ARGSORT LEAKY_RELU FLASH_ATTN_EXT FLASH_ATTN_BACK SSM_CONV SSM_SCAN WIN_PART WIN_UNPART GET_REL_POS ADD_REL_POS RWKV_WKV UNARY MAP_UNARY MAP_BINARY MAP_CUSTOM1_F32 MAP_CUSTOM2_F32 MAP_CUSTOM3_F32 MAP_CUSTOM1 MAP_CUSTOM2 MAP_CUSTOM3 CROSS_ENTROPY_LOSS CROSS_ENTROPY_LOSS_BACK OPT_STEP_ADAMW none x+y view(x,nb,offset)+=y->x x-y x*y x/y x^2 √x log(x) sin(x) cos(x) Σx Σx_k Σx/n argmax(x) count_equal(x) repeat(x) repeat_back(x) concat(x, y) silu_back(x) norm(x) rms_norm(x) rms_norm_back(x) group_norm(x) X*Y X[i]*Y x*v y-\>view(x) x-\>y cont(x) reshape(x) view(x) permute(x) transpose(x) get_rows(x) get_rows_back(x) diag(x) diag_mask_inf(x) diag_mask_zero(x) soft_max(x) soft_max_back(x) rope(x) rope_back(x) clamp(x) conv_transpose_1d(x) im2col(x) im2col_back(x) conv_transpose_2d(x) pool_1d(x) pool_2d(x) pool_2d_back(x) upscale(x) pad(x) arange(start, stop, step) timestep_embedding(timesteps, dim, max_period) argsort(x) leaky_relu(x) flash_attn_ext(x) flash_attn_back(x) ssm_conv(x) ssm_scan(x) win_part(x) win_unpart(x) get_rel_pos(x) add_rel_pos(x) rwkv_wkv(k, v, r, tf, td, s) unary(x) f(x) f(x,y) custom_f32(x) custom_f32(x,y) custom_f32(x,y,z) custom(x) custom(x,y) custom(x,y,z) cross_entropy_loss(x,y) cross_entropy_loss_back(x,y) adamw(x) ABS SGN NEG STEP TANH ELU RELU SIGMOID GELU GELU_QUICK SILU HARDSWISH HARDSIGMOID EXP Behavior may be unexpected when allocating 0 bytes for ggml_aligned_malloc!
unknown allocation error invalid alignment value insufficient memory %s: %s (attempted to allocate %6.2f MB)
ggml_aligned_malloc type >= 0 && type < GGML_TYPE_COUNT n_dims >= 1 && n_dims <= GGML_MAX_DIMS view_src == NULL || data_size == 0 || data_size + view_offs <= ggml_nbytes(view_src) %s: not enough space in the scratch memory pool (needed %zu, available %zu)
ggml_new_tensor_impl obj_new ggml_can_repeat_rows(b, a) ggml_is_quantized(a->type) || a->type == GGML_TYPE_F16 || a->type == GGML_TYPE_BF16 ggml_is_scalar(b) ggml_is_padded_1d(a) ggml_nelements(b) <= ggml_nelements(a) b->type == GGML_TYPE_F32 tensor != NULL ggml_nelements(a) >= ggml_nelements(b) offset < (size_t)(1 << 30) %s (copy of %s) mask->type == GGML_TYPE_F16 || mask->type == GGML_TYPE_F32 ggml_is_matrix(mask) mask->ne[0] == a->ne[0] mask->ne[1] >= a->ne[1] (mode & 1) == 0 && "mode & 1 == 1 is no longer supported" c->type == GGML_TYPE_F32 c->ne[0] >= n_dims / 2 a->ne[0] <= ne0 a->ne[1] <= ne1 a->ne[2] <= ne2 a->ne[3] <= ne3 ggml_are_same_shape(pw, ph) ggml_is_contiguous(pw) ggml_is_contiguous(ph) ph->type == GGML_TYPE_F32 pw->type == GGML_TYPE_F32 pw->ne[3] == a->ne[2] pw->ne[0]*pw->ne[0] == a->ne[0] pw->ne[1]*pw->ne[2] == a->ne[1] ggml_is_contiguous_1(a) n_tasks == GGML_N_TASKS_MAX || n_tasks > 0 Behavior may be unexpected when allocating 0 bytes for ggml_malloc!
%s: failed to allocate %6.2f MB
ggml_malloc Behavior may be unexpected when allocating 0 bytes for ggml_calloc!
ggml_calloc i != GGML_HASHSET_FULL replacements->set.keys[i] == NULL %s (clone) cgraph->nodes[cgraph->n_nodes - 1] == tensor cgraph->n_leafs < cgraph->size leaf_%d cgraph->n_nodes < cgraph->size node_%d /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/./ggml-impl.h src0->type == tensor->type tensor->grad->type == tensor->type !src1->grad || src1->grad->type == tensor->grad->type ggml_is_contiguous(src0->grad) ggml_is_contiguous(tensor->grad) offset % n0 == 0 nb1 % n0 == 0 nb2 % n0 == 0 nb3 % n0 == 0 (!src1 || !src1->grad) && "backward pass for softmax mask not implemented" (!src2 || !src2->grad) && "gradients for freq factors not implemented" FA backward pass not adapted after rework !src1->grad && "backward pass for labels not implemented" ggml_are_same_shape(tensor->src[i], tensor->src[i]->grad) %s: not enough space in the context's memory pool (needed %zu, available %zu)
ggml_new_object ((uintptr_t) (mem_buffer + obj_new->offs))%GGML_MEM_ALIGN == 0 %s: op not implemented: ggml_get_n_tasks %s
%d
rc == 0 warn: failed to set thread priority %d : %s (%d)
t == 0 || t == 1 ggml_nelements(dst) == ggml_nelements(src0) ggml_is_contiguous(dst) && ggml_is_contiguous(src0) ggml_can_repeat(src1, src0) && ggml_are_same_shape(src0, dst) nb0 == sizeof(float) nb00 == sizeof(float) ggml_are_same_shape(src0, src1) && ggml_are_same_shape(src0, dst) src0->type == GGML_TYPE_F16 dst->type == GGML_TYPE_F16 nb0 == sizeof(ggml_fp16_t) nb00 == sizeof(ggml_fp16_t) src1->type == GGML_TYPE_F32 dst->type == GGML_TYPE_BF16 nb0 == sizeof(ggml_bf16_t) nb00 == sizeof(ggml_bf16_t) nb00 == ggml_type_size(type) nb10 == sizeof(float) nb0 <= nb1 nb1 <= nb2 nb2 <= nb3 ggml_is_quantized(src0->type) ggml_are_same_shape(src0, dst) ggml_is_scalar(src1) dst->type == src0->type offset + (ne10 == 0 ? 0 : ne10-1)*nb0 + (ne11 == 0 ? 0 : ne11-1)*nb1 + (ne12 == 0 ? 0 : ne12-1)*nb2 + (ne13 == 0 ? 0 : ne13-1)*nb3 < ggml_nbytes(dst) offset + (ne10 == 0 ? 0 : ne10-1)*nb00 + (ne11 == 0 ? 0 : ne11-1)*nb01 + (ne12 == 0 ? 0 : ne12-1)*nb02 + (ne13 == 0 ? 0 : ne13-1)*nb03 < ggml_nbytes(src0) src1->type == GGML_TYPE_F32 && "only f32 src1 supported for now" dst->nb[0] == sizeof(float) src0->nb[0] == sizeof(float) ne0 == 1 ne1 == ne01 ne2 == ne02 ne3 == ne03 src1->type == GGML_TYPE_I32 ggml_are_same_shape(src0, src1) ggml_is_scalar(dst) dst->type == GGML_TYPE_I64 ggml_can_repeat(src0, dst) ggml_can_repeat(dst, src0) dim >= 0 && dim < 4 eps > 0.0f ggml_are_same_shape(src0, dst) && ggml_are_same_shape(src0, src1) ne0 == ne01 ne1 == ne11 ne2 == ne12 ne3 == ne13 nb10 == ggml_type_size(src1->type) ne02 == ne12 ne03 == ne13 ne0 == ne00 ne1 == ne10 dst->type == GGML_TYPE_F32 ggml_is_contiguous(src0) ggml_is_contiguous(dst) offset + im0*nb0 + im1*nb1 + im2*nb2 + im3*nb3 <= ggml_nbytes(dst) i01 >= 0 && i01 < ne01 dst->ne[0] == nc src0->nb[0] == sizeof(ggml_fp16_t) ne00 == ne0 ne00 == ne1 ne01 == 1 ne02 == ne2 ne03 == ne3 n_past >= 0 ggml_is_contiguous(src1) ggml_are_same_shape(src1, dst) n_dims <= ne0 n_dims % 2 == 0 src2->type == GGML_TYPE_F32 src2->ne[0] >= n_dims / 2 k0 == s0 false ggml_nelements(dst) == steps ne0 == D ne2 == N nbq0 == ggml_type_size(q->type) nbk0 == ggml_type_size(k->type) nbv0 == ggml_type_size(v->type) nek0 == D nev0 == D P >= 0 nbq0 == sizeof(float) nbk0 == sizeof(float) nbv0 == sizeof(float) nev1 == D ned0 == D ned1 == N ggml_blck_size(result_type) == 1 dst->ne[0] == nr src1->nb[0] == sizeof(float) src0->nb[1] == src0->ne[0]*sizeof(float) ggml_nelements(src1) + ggml_nelements(src0) == ggml_nelements(dst) src2->nb[0] == sizeof(float) src3->nb[0] == sizeof(float) src4->nb[0] == sizeof(float) src5->nb[0] == sizeof(float) src0->nb[2] == src0->ne[0]*src0->ne[1]*sizeof(float) src1->nb[3] == src1->ne[0]*src1->ne[1]*src1->ne[2]*sizeof(float) src0->nb[0] == ggml_type_size(src0->type) src1->nb[0] == ggml_type_size(src1->type) params->wsize >= sizeof(float) * (nth + nth * nc) ggml_is_contiguous(opt0) ggml_are_same_shape(src0, src0_grad) %-6s %-12s %8d %lld %lld %lld %lld %16zu %16zu %16zu %16zu %16p %32s
%-6s %-6s %-12s %8d %lld %lld %lld %lld %16zu %16zu %16zu %16zu %16p %32s
"%p":%s -> "%p":%s [ arrowhead = %s; style = %s; label = "%s"; ]
empty vee dashed solid "%p":%s -> "%p":%s [ label = "%s"; ]
ggml_is_scalar(f) f->type == GGML_TYPE_F32 np < GGML_MAX_PARAMS %s: invalid string length (%llu)
gguf_fread_str 0 <= type && type < GGUF_TYPE_COUNT info->n_dims <= GGML_MAX_DIMS 0 <= info->type && info->type < GGML_TYPE_COUNT info->ne[i] > 0 INT64_MAX/info->ne[1] > info->ne[0] INT64_MAX/info->ne[2] > info->ne[0]*info->ne[1] INT64_MAX/info->ne[3] > info->ne[0]*info->ne[1]*info->ne[2] u8 u16 u32 bool str arr u64 offset == info->offset %s: not enough space in the buffer to allocate %s (needed %zu, available %zu)
ggml_tallocr_alloc /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml-alloc.c not enough space in the buffer galloc != NULL galloc->bufts != NULL galloc->buffers != NULL galloc->buf_tallocs != NULL galloc->hash_set.keys != NULL galloc->hash_values != NULL galloc->node_allocs != NULL galloc->leaf_allocs != NULL %s: failed to allocate %s buffer of size %zu
ggml_gallocr_reserve_n buffer_id >= 0 && buffer_id < galloc->n_buffers ggml_get_no_alloc(ctx) == true %s: tensor %s is too large to fit in a %s buffer (tensor size: %zu, max buffer size: %zu)
ggml_backend_alloc_ctx_tensors_from_buft %s: not enough space in the buffer to allocate %zu bytes, largest block available %zu bytes
ggml_dyn_tallocr_alloc alloc->n_free_blocks < MAX_FREE_BLOCKS && "out of free blocks" /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml-backend.cpp base != NULL && "backend buffer base cannot be NULL" NULL tensor->data != NULL && "tensor not allocated" offset + size <= ggml_nbytes(tensor) && "tensor write out of bounds" offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds" buf != NULL && "tensor buffer not set" buf->iface.memset_tensor != NULL && "memset not supported by backend buffer" backend->iface.graph_plan_create != NULL backend->iface.graph_plan_free != NULL backend->iface.graph_plan_compute != NULL ggml_are_same_layout(src, dst) && "cannot copy tensors with different layouts" backend->iface.event_record != NULL event->device->iface.event_synchronize backend->iface.event_wait != NULL index < ggml_backend_reg_count() index < ggml_backend_dev_count() ggml_backend_is_cpu(backend_cpu) (uintptr_t)ptr % TENSOR_ALIGNMENT == 0 && "buffer pointer must be aligned" ctx->buffers != NULL ggml_backend_buffer_is_multi_buffer(buffer) n_backends > 0 n_backends <= GGML_SCHED_MAX_BACKENDS ggml_backend_is_cpu(backends[n_backends - 1]) GGML_SCHED_DEBUG ggml_backend_supports_buft(backends[b], sched->bufts[b]) (int)sched->hash_set.size >= measure_graph->n_nodes + measure_graph->n_leafs (int)sched->hash_set.size >= graph->n_nodes + graph->n_leafs i >= 0 && i < sched->n_backends backend_index >= 0 && backend_index < sched->n_backends tensor->buffer == NULL tensor->view_src != NULL tensor->view_src->buffer != NULL tensor->view_src->data != NULL tensor->data == NULL tensor->view_src == NULL addr >= ggml_backend_buffer_get_base(buffer) (char *)addr + ggml_backend_buffer_get_alloc_size(buffer, tensor) <= (char *)ggml_backend_buffer_get_base(buffer) + ggml_backend_buffer_get_size(buffer) failed to allocate context for graph copy
failed to allocate buffer for graph copy
vector CPU %s: failed to allocate buffer of size %zu
ggml_backend_cpu_buffer_type_alloc_buffer index == 0 machdep.cpu.brand_string ggml_backend_sched_split_graph sched->splits != NULL %s#%s#%d n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS sched->graph.nodes != NULL sched->graph.leafs != NULL pre-allocated tensor in a backend that cannot run the operation
## SPLIT #%d: %s # %d inputs: [%s (%5.5s)] node #%3d (%10.10s): %20.20s (%5.5s) [%5.5s %8.8s]: %20.20s (%5.5s) [%5.5s %8.8s] %zuM %zuK %s: failed to allocate graph
ggml_backend_sched_alloc_splits src != NULL src->data && "graph must be allocated" /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml-quants.c type == GGML_TYPE_IQ2_XXS || type == GGML_TYPE_IQ2_XS || type == GGML_TYPE_IQ1_S || type == GGML_TYPE_IQ1_M || type == GGML_TYPE_IQ2_S n_per_row%QK_K == 0 grid_size == 256 || grid_size == 512 n_per_row%QK4_NL == 0 k%QK4_NL == 0 %s: invalid type %d
ggml_validate_row_data %s: invalid size %zu for type %s (type size = %zu)
%s: found %d NaNs in row of %zu BF16 values
%s: found %d infinities in row of %zu BF16 values
quant_weights && "missing quantization weights" kgrid_q2xs && "forgot to call ggml_quantize_init()?" kmap_q2xs && "forgot to call ggml_quantize_init()?" kneighbors_q2xs && "forgot to call ggml_quantize_init()?" n%QK_K == 0 Oops: found point %u not on grid: %d scale >= 0 num_neighbors > 0 grid_index >= 0 kgrid_q3xs && "forgot to call ggml_quantize_init()?" kmap_q3xs && "forgot to call ggml_quantize_init()?" kneighbors_q3xs && "forgot to call ggml_quantize_init()?" besti1 >= 0 && besti2 >= 0 && best_shift != 0 Have %d neighbours
neighbour %d: sumqx = %g sumq2 = %g
besti1 >= 0 && besti2 >= 0 && best_k >= 0 ggml_validate_row_data: found inf value at block %zu
ggml_validate_row_data: found nan value at block %zu
Oops, did not find grid point %s: error: failed to allocate buffer, size = %8.2f MiB
ggml_backend_metal_buffer_from_ptr
%s: error: failed to allocate context
ggml_backend_metal_init /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml-metal.m ggml_backend_is_metal(backend) Metal ggml_backend_metal_buffer_type_alloc_buffer %s: error: vm_allocate failed
ggml_metal_host_malloc %s: allocated buffer, size = %8.2f MiB, (%8.2f / %8.2f)
ggml_backend_metal_log_allocated_size %s: warning: current allocated size is greater than the recommended max working set size
%s: allocating
ggml_metal_init %s: found device: %s
%s: picking default device: %s
ggml-metal default metallib %s: using embedded metal library
%s: error: %s
%s: GPU name: %s
%s: GPU family: MTLGPUFamilyApple%d (%d)
%s: GPU family: MTLGPUFamilyCommon%d (%d)
%s: GPU family: MTLGPUFamilyMetal%d (%d)
%s: simdgroup reduction support = %s
true %s: simdgroup matrix mul. support = %s
%s: hasUnifiedMemory = %s
%s: recommendedMaxWorkingSetSize = %8.2f MB
kernel_add %s: error: load pipeline error: %s
kernel_add_row kernel_sub kernel_sub_row kernel_mul kernel_mul_row kernel_div kernel_div_row kernel_repeat_f32 kernel_repeat_f16 kernel_repeat_i32 kernel_repeat_i16 kernel_scale kernel_scale_4 kernel_clamp kernel_tanh kernel_relu kernel_sigmoid kernel_gelu kernel_gelu_4 kernel_gelu_quick kernel_gelu_quick_4 kernel_silu kernel_silu_4 kernel_soft_max_f16 %s: skipping %-40s (not supported)
kernel_soft_max_f16_4 kernel_soft_max_f32 kernel_soft_max_f32_4 kernel_diag_mask_inf kernel_diag_mask_inf_8 kernel_get_rows_f32 kernel_get_rows_f16 kernel_get_rows_q4_0 kernel_get_rows_q4_1 kernel_get_rows_q5_0 kernel_get_rows_q5_1 kernel_get_rows_q8_0 kernel_get_rows_q2_K kernel_get_rows_q3_K kernel_get_rows_q4_K kernel_get_rows_q5_K kernel_get_rows_q6_K kernel_get_rows_iq2_xxs kernel_get_rows_iq2_xs kernel_get_rows_iq3_xxs kernel_get_rows_iq3_s kernel_get_rows_iq2_s kernel_get_rows_iq1_s kernel_get_rows_iq1_m kernel_get_rows_iq4_nl kernel_get_rows_iq4_xs kernel_get_rows_i32 kernel_rms_norm kernel_group_norm kernel_norm kernel_ssm_conv_f32 kernel_ssm_scan_f32 kernel_mul_mv_f32_f32 kernel_mul_mv_f16_f16 kernel_mul_mv_f16_f32 kernel_mul_mv_f16_f32_1row kernel_mul_mv_f16_f32_l4 kernel_mul_mv_q4_0_f32 kernel_mul_mv_q4_1_f32 kernel_mul_mv_q5_0_f32 kernel_mul_mv_q5_1_f32 kernel_mul_mv_q8_0_f32 kernel_mul_mv_q2_K_f32 kernel_mul_mv_q3_K_f32 kernel_mul_mv_q4_K_f32 kernel_mul_mv_q5_K_f32 kernel_mul_mv_q6_K_f32 kernel_mul_mv_iq2_xxs_f32 kernel_mul_mv_iq2_xs_f32 kernel_mul_mv_iq3_xxs_f32 kernel_mul_mv_iq3_s_f32 kernel_mul_mv_iq2_s_f32 kernel_mul_mv_iq1_s_f32 kernel_mul_mv_iq1_m_f32 kernel_mul_mv_iq4_nl_f32 kernel_mul_mv_iq4_xs_f32 kernel_mul_mv_id_f32_f32 kernel_mul_mv_id_f16_f32 kernel_mul_mv_id_q4_0_f32 kernel_mul_mv_id_q4_1_f32 kernel_mul_mv_id_q5_0_f32 kernel_mul_mv_id_q5_1_f32 kernel_mul_mv_id_q8_0_f32 kernel_mul_mv_id_q2_K_f32 kernel_mul_mv_id_q3_K_f32 kernel_mul_mv_id_q4_K_f32 kernel_mul_mv_id_q5_K_f32 kernel_mul_mv_id_q6_K_f32 kernel_mul_mv_id_iq2_xxs_f32 kernel_mul_mv_id_iq2_xs_f32 kernel_mul_mv_id_iq3_xxs_f32 kernel_mul_mv_id_iq3_s_f32 kernel_mul_mv_id_iq2_s_f32 kernel_mul_mv_id_iq1_s_f32 kernel_mul_mv_id_iq1_m_f32 kernel_mul_mv_id_iq4_nl_f32 kernel_mul_mv_id_iq4_xs_f32 kernel_mul_mm_f32_f32 kernel_mul_mm_f16_f32 kernel_mul_mm_q4_0_f32 kernel_mul_mm_q4_1_f32 kernel_mul_mm_q5_0_f32 kernel_mul_mm_q5_1_f32 kernel_mul_mm_q8_0_f32 kernel_mul_mm_q2_K_f32 kernel_mul_mm_q3_K_f32 kernel_mul_mm_q4_K_f32 kernel_mul_mm_q5_K_f32 kernel_mul_mm_q6_K_f32 kernel_mul_mm_iq2_xxs_f32 kernel_mul_mm_iq2_xs_f32 kernel_mul_mm_iq3_xxs_f32 kernel_mul_mm_iq3_s_f32 kernel_mul_mm_iq2_s_f32 kernel_mul_mm_iq1_s_f32 kernel_mul_mm_iq1_m_f32 kernel_mul_mm_iq4_nl_f32 kernel_mul_mm_iq4_xs_f32 kernel_mul_mm_id_f32_f32 kernel_mul_mm_id_f16_f32 kernel_mul_mm_id_q4_0_f32 kernel_mul_mm_id_q4_1_f32 kernel_mul_mm_id_q5_0_f32 kernel_mul_mm_id_q5_1_f32 kernel_mul_mm_id_q8_0_f32 kernel_mul_mm_id_q2_K_f32 kernel_mul_mm_id_q3_K_f32 kernel_mul_mm_id_q4_K_f32 kernel_mul_mm_id_q5_K_f32 kernel_mul_mm_id_q6_K_f32 kernel_mul_mm_id_iq2_xxs_f32 kernel_mul_mm_id_iq2_xs_f32 kernel_mul_mm_id_iq3_xxs_f32 kernel_mul_mm_id_iq3_s_f32 kernel_mul_mm_id_iq2_s_f32 kernel_mul_mm_id_iq1_s_f32 kernel_mul_mm_id_iq1_m_f32 kernel_mul_mm_id_iq4_nl_f32 kernel_mul_mm_id_iq4_xs_f32 kernel_rope_norm_f32 kernel_rope_norm_f16 kernel_rope_neox_f32 kernel_rope_neox_f16 kernel_im2col_f16 kernel_im2col_f32 kernel_upscale_f32 kernel_pad_f32 kernel_timestep_embedding_f32 kernel_arange_f32 kernel_argsort_f32_i32_asc kernel_argsort_f32_i32_desc kernel_leaky_relu_f32 kernel_flash_attn_ext_f16_h64 kernel_flash_attn_ext_f16_h80 kernel_flash_attn_ext_f16_h96 kernel_flash_attn_ext_f16_h112 kernel_flash_attn_ext_f16_h128 kernel_flash_attn_ext_vec_f16_h128 kernel_cpy_f32_f16 kernel_cpy_f32_f32 kernel_cpy_f16_f16 kernel_cpy_f16_f32 kernel_cpy_f32_q8_0 kernel_cpy_f32_q4_0 kernel_cpy_f32_q4_1 kernel_cpy_f32_q5_0 kernel_cpy_f32_q5_1 kernel_cpy_f32_iq4_nl kernel_concat kernel_sqr kernel_sqrt kernel_sin kernel_cos kernel_sum_rows %s: deallocating
ggml_metal_free /tmp/perf-metal.gputrace %s: error: unable to start capture '%s'
ggml_metal_graph_compute v16@?0Q8 %s: command buffer %d failed with status %lu
error: %s
%s: command buffer %d aborted %s: error: unsupported op '%s'
ggml_metal_encode_node unsupported op src0t == GGML_TYPE_F32 src1t == GGML_TYPE_F32 ne11 == 1 dstt == GGML_TYPE_F32 %s: node %3d, op = %8s not implemented
!src1 || src1->type == GGML_TYPE_F16 || src1->type == GGML_TYPE_F32 src3 src4 src5 ne00 == ne10 ne12 % ne02 == 0 ne13 % ne03 == 0 nb01 % 16 == 0 nb01 % 8 == 0 MUL MAT-MAT not implemented Asserting on type %d
not implemented src2t == GGML_TYPE_I32 !ggml_is_transposed(src0) !ggml_is_transposed(src1) dst_rows <= dst_rows_max MUL_MAT_ID not implemented ne00 >= nth0*nth1 ne00 % 4 == 0 ggml_is_contiguous_1(src0) ne10 == ne02 dst->type == GGML_TYPE_F16 || dst->type == GGML_TYPE_F32 src0->type == GGML_TYPE_F32 dst->type == GGML_TYPE_I32 ne11 % 32 == 0 ggml_are_same_shape (src1, src2) !src3 || src3->type == GGML_TYPE_F16 !src3 || src3->ne[1] >= GGML_PAD(src0->ne[1], 8) && "the Flash-Attention Metal kernel requires the mask to be padded to 8 and at least n_queries big" unsupported size: %lld
add template specialization for this size
add template specialization for this size smem <= ctx->device.maxThreadgroupMemoryLength ne00 % ggml_blck_size(src0->type) == 0 ne0 % ggml_blck_size(dst->type) == 0 %s: error: node %3d, op = %8s not implemented
%s: error: tensor '%s' buffer is nil
ggml_metal_get_buffer /Users/runner/work/whisper-jni/whisper-jni/build/w/src/whisper-cpp/ggml/src/ggml-blas.cpp ggml_backend_is_blas(backend_blas) BLAS %s: unsupported op %s
ggml_backend_blas_graph_compute thread constructor failed ??
T T? c\ ??
T T? c\ ?? U? d\ ??
a a ? m? ?? U? d\ ??
a a ? m? ??
z? ?: ?? ? ? ? ?) ?? ? ?? ?: ?? c c ? o? ??+ &* ?? 8 ") ??
z? ?: ?? ? ? ? ?) ??
z? ?: ?? Mv m" ??md ?
?
? ?
? ? ? ?
? ?? ?? ? ? ?? ?? ?? ?? ?? ?? ? ? ? ? ? ?? 8 ") ??
??-"Kp]ne ? ? ? ??? ??1u CF GGMLMetalClass bundleForClass: beginScope maxBufferLength commit pushDebugGroup: countByEnumeratingWithState:objects:count: supportsFamily: UTF8String enqueue commandBufferWithUnretainedReferences newCommandQueue computeCommandEncoder setDestination: currentAllocatedSize description dictionary dispatchThreadgroups:threadsPerThreadgroup: endEncoding endScope error fileURLWithPath: hasUnifiedMemory initWithBytes:length:encoding: localizedDescription maxThreadgroupMemoryLength maxTotalThreadsPerThreadgroup name newBufferWithBytesNoCopy:length:options:deallocator: newCaptureScopeWithDevice: newComputePipelineStateWithFunction:error: newFunctionWithName: newLibraryWithSource:options:error: pathForResource:ofType: popDebugGroup recommendedMaxWorkingSetSize setBuffer:offset:atIndex: setBytes:length:atIndex: setCaptureObject: setComputePipelineState: setOutputURL: setPreprocessorMacros: setThreadgroupMemoryLength:atIndex: sharedCaptureManager startCaptureWithDescriptor:error: status stopCapture stringWithCString:encoding: stringWithFormat: waitUntilCompleted < @ ?X! a Q! QX
?< d ? `? ?? ? ?? 0 ?? ? ?? ? ? ? ? (? @ 8? p
L? ? `? ?b t? ?c ?? ?e ?? ?h ?? ?i ?? ?k ?? pU ?? h P? @h `? i ?? ? ? p ? ? ? ` P! @' ?( ) P2 ?2 < 0< ?> PP Q ?Q ?Q 0R ?R ?R ?T ?T V V pW ?W PZ ?Z P\ p\ @^ `^ 0` @` ?` a @a ?a b ?b ?b @c ?c d ?e Pf ?h @j 0k l 0l pm `n pn `o po `p pp `q pq `r pr `s ps @t 0u @u 0v @v 0w @w 0x @x ?x P{ `{ ?{ `| ?| 0~ ?~ ? @? p? ?? ?? ?? ? ?? ?? ?? @? ? ?? P? 0? @? ? 0? О 0? ?? ? ?? Р `? ?? Ф `? ?? ?? ?? ?? p? ? Я ?? P? ?? ?? ? ? ?? к ? ? ?? @? н ? ?? ?? ? ? ? ?? ?? ?? ? ? ?? 0? ?? ?? ? ? ?? ?? ?? @? ?? ?? p? 0? ? ?? ? ?? ?? ?? P? ? ?*?*?+ ?, / ?/?/?1 3?3p4`= ?? ?!PJ PK?O ? ??@? ??@? ????`?б @?? P??? ?????P? ??@? P?????`??????? ??????? Л? ?? ?P?0? Ю0? @?p? `??? ?`? ?@? p?P? 0?`? @?но ????0?????P? ?`? ???p? ??? ??p? ??p? ??`???? 0?`??? ????@?p??? P P? ? ? ?P?$?$@&?&?& ?&0'?' P(?( ?( )?)?*?* ?*@+ ,?,?- ?-?. /?/@0 4@6 0R?d ?hz zp? ??? ??
?
@4?9 ?B?} ?}?~ ? ? P? ? ?? ????
?O ?O `P 0g Pg `? ?? ?? ? `
`
?
?!
?!
"
"
?'
p(
@z
`z
?z
@{
?{
?|
?|
@}
?}
?}
~
?~
?~
?
P?
? @ ? ? ? 0 ? ? ?) ?* ?* p+ ?+ `, ?, ?0 `5 ?t ?XQ ?< ¥ ?$ ƥ @ ?@ ?@ & ʥ Ў ?w ?r ? ϥ @? p{ ?w ?? ԥ $ ԥ $ ߥ ? ?? ?{ ?? ? @? 0? ?? ? ? " p? `? @? ?q ? ? $ @? ?? ? T 0? ?? ? ? ?? n ?? ? 0? ?
?? ? p
? @? ?
? ? ?A G ?* ? ? ? @v ` `] ? ? $ ? ? B p? ? ? J 0? ? ? b 0? У ? ` (? 2 ?? ?& .? ?? ?
?
?- 5? n ?? ? ? p# ;? R ?? ?/
p/
? A? ? @? ?
p
?1 H? K? O? S? W? [? 8 P? `) a? 0B ?D ?C P' f? 0{ @? o? ?? x? 0? 0? ?? 6 @? `? ? ?? B ?? p? ? ? k? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? æ ̦ Ѧ ئ ? ? ?? ?? ?
? ? $? /? 7? B? K? Q? U? Y? ^? f? k? s? }? ?? ?? ?? ?? ?? ?? ͧ ҧ ܧ ? ?? ?? ? ? !? )? 6? >? B? I? \? d? o? ~? ?? ?? ?? ?? ?? ?? ̨ ը ۨ ? ? ? ? ? ,? 8? D? W? o? ~? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? © ɩ ͩ ө ٩ ? ? ?? ? ? %? -? 9? J? X? \? X? c? g? s? y? ?? ?? ?? ?? ?? ?? ɪ Ѫ ? ?? ? ? ? &? /? D? N? ]? r? }? ?? ?? ?? ?? ī ? ?? ? ? 1? =? I? U? c? r? ?? ?? ?? ?? ?? ?? Ѭ ? ?? ?? ? ?
typedef uint16_t ggml_half;
typedef uint32_t ggml_half2;
#define GGML_COMMON_AGGR
#define GGML_COMMON_DECL
#elif defined(GGML_COMMON_DECL_METAL)
#include
typedef half ggml_half;
typedef half2 ggml_half2;
#define GGML_COMMON_AGGR
#define GGML_COMMON_DECL
#elif defined(GGML_COMMON_DECL_CUDA)
#if defined(GGML_COMMON_DECL_MUSA)
#include
#else
#include
#endif
#include
typedef half ggml_half;
typedef half2 ggml_half2;
#define GGML_COMMON_AGGR data
#define GGML_COMMON_DECL
#elif defined(GGML_COMMON_DECL_HIP)
#include
#include
typedef half ggml_half;
typedef half2 ggml_half2;
#define GGML_COMMON_AGGR data
#define GGML_COMMON_DECL
#elif defined(GGML_COMMON_DECL_SYCL)
#include
#include
typedef sycl::half ggml_half;
typedef sycl::half2 ggml_half2;
#define GGML_COMMON_AGGR data
#define GGML_COMMON_DECL
#endif
#if defined(GGML_COMMON_DECL)
#ifndef __cplusplus
#ifndef static_assert
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201100L)
#define static_assert(cond, msg) _Static_assert(cond, msg)
#else
#define static_assert(cond, msg) struct global_scope_noop_trick
#endif
#endif
#endif // __cplusplus
// QK = number of values after dequantization
// QK_K = super-block size
#define QK_K 256
#define K_SCALE_SIZE 12
#if defined(GGML_COMMON_DECL_CUDA) || defined(GGML_COMMON_DECL_HIP) || defined(GGML_COMMON_DECL_SYCL)
// QR = QK / number of values before dequantization
// QI = number of 32 bit integers before dequantization
#define QI4_0 (QK4_0 / (4 * QR4_0))
#define QR4_0 2
#define QI4_1 (QK4_1 / (4 * QR4_1))
#define QR4_1 2
#define QI5_0 (QK5_0 / (4 * QR5_0))
#define QR5_0 2
#define QI5_1 (QK5_1 / (4 * QR5_1))
#define QR5_1 2
#define QI8_0 (QK8_0 / (4 * QR8_0))
#define QR8_0 1
#define QI8_1 (QK8_1 / (4 * QR8_1))
#define QR8_1 1
#define QI2_K (QK_K / (4*QR2_K))
#define QR2_K 4
#define QI3_K (QK_K / (4*QR3_K))
#define QR3_K 4
#define QI4_K (QK_K / (4*QR4_K))
#define QR4_K 2
#define QI5_K (QK_K / (4*QR5_K))
#define QR5_K 2
#define QI6_K (QK_K / (4*QR6_K))
#define QR6_K 2
#define QI2_XXS (QK_K / (4*QR2_XXS))
#define QR2_XXS 4
#define QI2_XS (QK_K / (4*QR2_XS))
#define QR2_XS 4
#define QI2_S (QK_K / (4*QR2_S))
#define QR2_S 4
#define QI3_XXS (QK_K / (4*QR3_XXS))
#define QR3_XXS 4
#define QI3_XS (QK_K / (4*QR3_XS))
#define QR3_XS 4
#define QI1_S (QK_K / (4*QR1_S))
#define QR1_S 8
#define QI1_M (QK_K / (4*QR1_M))
#define QR1_M 8
#define QI4_NL (QK4_NL / (4*QR4_NL))
#define QR4_NL 2
#define QI4_XS (QK_K / (4*QR4_XS))
#define QR4_XS 2
#define QI3_S (QK_K / (4*QR3_S))
#define QR3_S 4
#endif // GGML_COMMON_DECL_CUDA || GGML_COMMON_DECL_HIP
#define QK4_0 32
typedef struct {
ggml_half d; // delta
uint8_t qs[QK4_0 / 2]; // nibbles / quants
} block_q4_0;
static_assert(sizeof(block_q4_0) == sizeof(ggml_half) + QK4_0 / 2, "wrong q4_0 block size/padding");
#define QK4_1 32
typedef struct {
union {
struct {
ggml_half d; // delta
ggml_half m; // min
} GGML_COMMON_AGGR;
ggml_half2 dm;
};
uint8_t qs[QK4_1 / 2]; // nibbles / quants
} block_q4_1;
static_assert(sizeof(block_q4_1) == 2 * sizeof(ggml_half) + QK4_1 / 2, "wrong q4_1 block size/padding");
#define QK5_0 32
typedef struct {
ggml_half d; // delta
uint8_t qh[4]; // 5-th bit of quants
uint8_t qs[QK5_0 / 2]; // nibbles / quants
} block_q5_0;
static_assert(sizeof(block_q5_0) == sizeof(ggml_half) + sizeof(uint32_t) + QK5_0 / 2, "wrong q5_0 block size/padding");
#define QK5_1 32
typedef struct {
union {
struct {
ggml_half d; // delta
ggml_half m; // min
} GGML_COMMON_AGGR;
ggml_half2 dm;
};
uint8_t qh[4]; // 5-th bit of quants
uint8_t qs[QK5_1 / 2]; // nibbles / quants
} block_q5_1;
static_assert(sizeof(block_q5_1) == 2 * sizeof(ggml_half) + sizeof(uint32_t) + QK5_1 / 2, "wrong q5_1 block size/padding");
#define QK8_0 32
typedef struct {
ggml_half d; // delta
int8_t qs[QK8_0]; // quants
} block_q8_0;
static_assert(sizeof(block_q8_0) == sizeof(ggml_half) + QK8_0, "wrong q8_0 block size/padding");
#define QK8_1 32
typedef struct {
union {
struct {
ggml_half d; // delta
ggml_half s; // d * sum(qs[i])
} GGML_COMMON_AGGR;
ggml_half2 ds;
};
int8_t qs[QK8_1]; // quants
} block_q8_1;
static_assert(sizeof(block_q8_1) == 2*sizeof(ggml_half) + QK8_1, "wrong q8_1 block size/padding");
typedef struct {
ggml_half d[4]; // deltas for 4 q4_0 blocks
uint8_t qs[QK4_0 * 2]; // nibbles / quants for 4 q4_0 blocks
} block_q4_0x4;
static_assert(sizeof(block_q4_0x4) == 4 * sizeof(ggml_half) + QK4_0 * 2, "wrong q4_0x4 block size/padding");
typedef struct {
ggml_half d[8]; // deltas for 8 q4_0 blocks
uint8_t qs[QK4_0 * 4]; // nibbles / quants for 8 q4_0 blocks
} block_q4_0x8;
static_assert(sizeof(block_q4_0x8) == 8 * sizeof(ggml_half) + QK4_0 * 4, "wrong q4_0x8 block size/padding");
typedef struct {
ggml_half d[4]; // deltas for 4 q8_0 blocks
int8_t qs[QK8_0 * 4]; // quants for 4 q8_0 blocks
} block_q8_0x4;
static_assert(sizeof(block_q8_0x4) == 4 * sizeof(ggml_half) + QK8_0 * 4, "wrong q8_0x4 block size/padding");
typedef struct {
ggml_half d[8]; // deltas for 8 q8_0 blocks
int8_t qs[QK8_0 * 8]; // quants for 8 q8_0 blocks
} block_q8_0x8;
static_assert(sizeof(block_q8_0x8) == 8 * sizeof(ggml_half) + QK8_0 * 8, "wrong q8_0x8 block size/padding");
//
// Ternary quantization
//
// 1.6875 bpw
typedef struct {
uint8_t qs[(QK_K - 4 * QK_K / 64) / 5]; // 5 elements per byte (3^5 = 243 < 256)
uint8_t qh[QK_K/64]; // 4 elements per byte
ggml_half d;
} block_tq1_0;
static_assert(sizeof(block_tq1_0) == sizeof(ggml_half) + QK_K / 64 + (QK_K - 4 * QK_K / 64) / 5, "wrong tq1_0 block size/padding");
// 2.0625 bpw
typedef struct {
uint8_t qs[QK_K/4]; // 2 bits per element
ggml_half d;
} block_tq2_0;
static_assert(sizeof(block_tq2_0) == sizeof(ggml_half) + QK_K / 4, "wrong tq2_0 block size/padding");
//
// Super-block quantization structures
//
// 2-bit quantization
// weight is represented as x = a * q + b
// 16 blocks of 16 elements each
// Effectively 2.625 bits per weight
typedef struct {
uint8_t scales[QK_K/16]; // scales and mins, quantized with 4 bits
uint8_t qs[QK_K/4]; // quants
union {
struct {
ggml_half d; // super-block scale for quantized scales
ggml_half dmin; // super-block scale for quantized mins
} GGML_COMMON_AGGR;
ggml_half2 dm;
};
} block_q2_K;
static_assert(sizeof(block_q2_K) == 2*sizeof(ggml_half) + QK_K/16 + QK_K/4, "wrong q2_K block size/padding");
// 3-bit quantization
// weight is represented as x = a * q
// 16 blocks of 16 elements each
// Effectively 3.4375 bits per weight
typedef struct {
uint8_t hmask[QK_K/8]; // quants - high bit
uint8_t qs[QK_K/4]; // quants - low 2 bits
uint8_t scales[12]; // scales, quantized with 6 bits
ggml_half d; // super-block scale
} block_q3_K;
static_assert(sizeof(block_q3_K) == sizeof(ggml_half) + QK_K / 4 + QK_K / 8 + 12, "wrong q3_K block size/padding");
// 4-bit quantization
// 8 blocks of 32 elements each
// weight is represented as x = a * q + b
// Effectively 4.5 bits per weight
typedef struct {
union {
struct {
ggml_half d; // super-block scale for quantized scales
ggml_half dmin; // super-block scale for quantized mins
} GGML_COMMON_AGGR;
ggml_half2 dm;
};
uint8_t scales[K_SCALE_SIZE]; // scales and mins, quantized with 6 bits
uint8_t qs[QK_K/2]; // 4--bit quants
} block_q4_K;
static_assert(sizeof(block_q4_K) == 2*sizeof(ggml_half) + K_SCALE_SIZE + QK_K/2, "wrong q4_K block size/padding");
// 5-bit quantization
// 8 blocks of 32 elements each
// weight is represented as x = a * q + b
// Effectively 5.5 bits per weight
typedef struct {
union {
struct {
ggml_half d; // super-block scale for quantized scales
ggml_half dmin; // super-block scale for quantized mins
} GGML_COMMON_AGGR;
ggml_half2 dm;
};
uint8_t scales[K_SCALE_SIZE]; // scales and mins, quantized with 6 bits
uint8_t qh[QK_K/8]; // quants, high bit
uint8_t qs[QK_K/2]; // quants, low 4 bits
} block_q5_K;
static_assert(sizeof(block_q5_K) == 2*sizeof(ggml_half) + K_SCALE_SIZE + QK_K/2 + QK_K/8, "wrong q5_K block size/padding");
// 6-bit quantization
// weight is represented as x = a * q
// 16 blocks of 16 elements each
// Effectively 6.5625 bits per weight
typedef struct {
uint8_t ql[QK_K/2]; // quants, lower 4 bits
uint8_t qh[QK_K/4]; // quants, upper 2 bits
int8_t scales[QK_K/16]; // scales, quantized with 8 bits
ggml_half d; // super-block scale
} block_q6_K;
static_assert(sizeof(block_q6_K) == sizeof(ggml_half) + QK_K / 16 + 3*QK_K/4, "wrong q6_K block size/padding");
// This is only used for intermediate quantization and dot products
typedef struct {
float d; // delta
int8_t qs[QK_K]; // quants
int16_t bsums[QK_K/16]; // sum of quants in groups of 16
} block_q8_K;
static_assert(sizeof(block_q8_K) == sizeof(float) + QK_K + QK_K/16*sizeof(int16_t), "wrong q8_K block size/padding");
// (Almost) "true" 2-bit quantization.
// Due to the need to use blocks as per ggml design, it ends up using
// 2.0625 bpw because of the 16-bit scale for each block of 256.
typedef struct {
ggml_half d;
uint16_t qs[QK_K/8];
} block_iq2_xxs;
static_assert(sizeof(block_iq2_xxs) == sizeof(ggml_half) + QK_K/8*sizeof(uint16_t), "wrong iq2_xxs block size/padding");
// 2.3125 bpw quants
typedef struct {
ggml_half d;
uint16_t qs[QK_K/8];
uint8_t scales[QK_K/32];
} block_iq2_xs;
static_assert(sizeof(block_iq2_xs) == sizeof(ggml_half) + QK_K/8*sizeof(uint16_t) + QK_K/32, "wrong iq2_xs block size/padding");
// 2.5625 bpw quants
typedef struct {
ggml_half d;
uint8_t qs[QK_K/4];
uint8_t qh[QK_K/32];
uint8_t scales[QK_K/32];
} block_iq2_s;
static_assert(sizeof(block_iq2_s) == sizeof(ggml_half) + QK_K/4 + QK_K/16, "wrong iq2_s block size/padding");
// (Almost) "true" 3-bit quantization.
// Due to the need to use blocks as per ggml design, it ends up using
// 3.0625 bpw because of the 16-bit scale for each block of 256.
typedef struct {
ggml_half d;
uint8_t qs[3*QK_K/8];
} block_iq3_xxs;
static_assert(sizeof(block_iq3_xxs) == sizeof(ggml_half) + 3*(QK_K/8), "wrong iq3_xxs block size/padding");
// 3.4375 bpw
#define IQ3S_N_SCALE QK_K/64
typedef struct {
ggml_half d;
uint8_t qs[QK_K/4];
uint8_t qh[QK_K/32];
uint8_t signs[QK_K/8];
uint8_t scales[IQ3S_N_SCALE];
} block_iq3_s;
static_assert(sizeof(block_iq3_s) == sizeof(ggml_half) + 13*(QK_K/32) + IQ3S_N_SCALE, "wrong iq3_s block size/padding");
// 1.5625 bpw
typedef struct {
ggml_half d;
uint8_t qs[QK_K/8];
uint16_t qh[QK_K/32];
} block_iq1_s;
static_assert(sizeof(block_iq1_s) == sizeof(ggml_half) + QK_K/8 + QK_K/16, "wrong iq1_s block size/padding");
// 1.75 bpw
typedef struct {
uint8_t qs[QK_K/8]; // grid index, low 8 bits
uint8_t qh[QK_K/16]; // grid index, high 3 bits + grid shift bit (for two groups of 8)
uint8_t scales[QK_K/32]; // 3-bit block scales (4-bit if QK_K == 64)
} block_iq1_m;
static_assert(sizeof(block_iq1_m) == QK_K/8 + QK_K/16 + QK_K/32, "wrong iq1_m block size/padding");
// Used by IQ1_M quants
typedef union {
ggml_half f16;
uint16_t u16;
} iq1m_scale_t;
// Non-linear quants
#define QK4_NL 32
typedef struct {
ggml_half d;
uint8_t qs[QK4_NL/2];
} block_iq4_nl;
static_assert(sizeof(block_iq4_nl) == sizeof(ggml_half) + QK4_NL/2, "wrong iq4_nl block size/padding");
typedef struct {
ggml_half d;
uint16_t scales_h;
uint8_t scales_l[QK_K/64];
uint8_t qs[QK_K/2];
} block_iq4_xs;
static_assert(sizeof(block_iq4_xs) == sizeof(ggml_half) + sizeof(uint16_t) + QK_K/64 + QK_K/2, "wrong iq4_xs block size/padding");
#endif // GGML_COMMON_DECL
#endif // GGML_COMMON_DECL
////////////////////////////////////////////////////////////////////////////////
#ifndef GGML_COMMON_IMPL
#if defined(GGML_COMMON_IMPL_C)
#include
#define GGML_TABLE_BEGIN(type, name, size) static const type name[size] = {
#define GGML_TABLE_END() };
#define GGML_COMMON_IMPL
#elif defined(GGML_COMMON_IMPL_METAL)
#include
#define GGML_TABLE_BEGIN(type, name, size) static const constant type name[size] = {
#define GGML_TABLE_END() };
#define GGML_COMMON_IMPL
#elif defined(GGML_COMMON_IMPL_CUDA) || defined(GGML_COMMON_IMPL_HIP) || defined(GGML_COMMON_IMPL_MUSA)
#include
#define GGML_TABLE_BEGIN(type, name, size) static const __device__ type name[size] = {
#define GGML_TABLE_END() };
#define GGML_COMMON_IMPL
#elif defined(GGML_COMMON_IMPL_SYCL)
#include
#define GGML_TABLE_BEGIN(type, name, size) static const type name[size] = {
#define GGML_TABLE_END() };
#define GGML_COMMON_IMPL
#endif
#if defined(GGML_COMMON_IMPL)
GGML_TABLE_BEGIN(uint8_t, kmask_iq2xs, 8)
1, 2, 4, 8, 16, 32, 64, 128
GGML_TABLE_END()
GGML_TABLE_BEGIN(uint8_t, ksigns_iq2xs, 128)
0, 129, 130, 3, 132, 5, 6, 135, 136, 9, 10, 139, 12, 141, 142, 15,
144, 17, 18, 147, 20, 149, 150, 23, 24, 153, 154, 27, 156, 29, 30, 159,
160, 33, 34, 163, 36, 165, 166, 39, 40, 169, 170, 43, 172, 45, 46, 175,
48, 177, 178, 51, 180, 53, 54, 183, 184, 57, 58, 187, 60, 189, 190, 63,
192, 65, 66, 195, 68, 197, 198, 71, 72, 201, 202, 75, 204, 77, 78, 207,
80, 209, 210, 83, 212, 85, 86, 215, 216, 89, 90, 219, 92, 221, 222, 95,
96, 225, 226, 99, 228, 101, 102, 231, 232, 105, 106, 235, 108, 237, 238, 111,
240, 113, 114, 243, 116, 245, 246, 119, 120, 249, 250, 123, 252, 125, 126, 255,
GGML_TABLE_END()
//#if __CUDA_ARCH__ >= MIN_CC_DP4A // lowest compute capability for integer intrinsics
GGML_TABLE_BEGIN(uint64_t, ksigns64, 128)
0x0000000000000000, 0xff000000000000ff, 0xff0000000000ff00, 0x000000000000ffff,
0xff00000000ff0000, 0x0000000000ff00ff, 0x0000000000ffff00, 0xff00000000ffffff,
0xff000000ff000000, 0x00000000ff0000ff, 0x00000000ff00ff00, 0xff000000ff00ffff,
0x00000000ffff0000, 0xff000000ffff00ff, 0xff000000ffffff00, 0x00000000ffffffff,
0xff0000ff00000000, 0x000000ff000000ff, 0x000000ff0000ff00, 0xff0000ff0000ffff,
0x000000ff00ff0000, 0xff0000ff00ff00ff, 0xff0000ff00ffff00, 0x000000ff00ffffff,
0x000000ffff000000, 0xff0000ffff0000ff, 0xff0000ffff00ff00, 0x000000ffff00ffff,
0xff0000ffffff0000, 0x000000ffffff00ff, 0x000000ffffffff00, 0xff0000ffffffffff,
0xff00ff0000000000, 0x0000ff00000000ff, 0x0000ff000000ff00, 0xff00ff000000ffff,
0x0000ff0000ff0000, 0xff00ff0000ff00ff, 0xff00ff0000ffff00, 0x0000ff0000ffffff,
0x0000ff00ff000000, 0xff00ff00ff0000ff, 0xff00ff00ff00ff00, 0x0000ff00ff00ffff,
0xff00ff00ffff0000, 0x0000ff00ffff00ff, 0x0000ff00ffffff00, 0xff00ff00ffffffff,
0x0000ffff00000000, 0xff00ffff000000ff, 0xff00ffff0000ff00, 0x0000ffff0000ffff,
0xff00ffff00ff0000, 0x0000ffff00ff00ff, 0x0000ffff00ffff00, 0xff00ffff00ffffff,
0xff00ffffff000000, 0x0000ffffff0000ff, 0x0000ffffff00ff00, 0xff00ffffff00ffff,
0x0000ffffffff0000, 0xff00ffffffff00ff, 0xff00ffffffffff00, 0x0000ffffffffffff,
0xffff000000000000, 0x00ff0000000000ff, 0x00ff00000000ff00, 0xffff00000000ffff,
0x00ff000000ff0000, 0xffff000000ff00ff, 0xffff000000ffff00, 0x00ff000000ffffff,
0x00ff0000ff000000, 0xffff0000ff0000ff, 0xffff0000ff00ff00, 0x00ff0000ff00ffff,
0xffff0000ffff0000, 0x00ff0000ffff00ff, 0x00ff0000ffffff00, 0xffff0000ffffffff,
0x00ff00ff00000000, 0xffff00ff000000ff, 0xffff00ff0000ff00, 0x00ff00ff0000ffff,
0xffff00ff00ff0000, 0x00ff00ff00ff00ff, 0x00ff00ff00ffff00, 0xffff00ff00ffffff,
0xffff00ffff000000, 0x00ff00ffff0000ff, 0x00ff00ffff00ff00, 0xffff00ffff00ffff,
0x00ff00ffffff0000, 0xffff00ffffff00ff, 0xffff00ffffffff00, 0x00ff00ffffffffff,
0x00ffff0000000000, 0xffffff00000000ff, 0xffffff000000ff00, 0x00ffff000000ffff,
0xffffff0000ff0000, 0x00ffff0000ff00ff, 0x00ffff0000ffff00, 0xffffff0000ffffff,
0xffffff00ff000000, 0x00ffff00ff0000ff, 0x00ffff00ff00ff00, 0xffffff00ff00ffff,
0x00ffff00ffff0000, 0xffffff00ffff00ff, 0xffffff00ffffff00, 0x00ffff00ffffffff,
0xffffffff00000000, 0x00ffffff000000ff, 0x00ffffff0000ff00, 0xffffffff0000ffff,
0x00ffffff00ff0000, 0xffffffff00ff00ff, 0xffffffff00ffff00, 0x00ffffff00ffffff,
0x00ffffffff000000, 0xffffffffff0000ff, 0xffffffffff00ff00, 0x00ffffffff00ffff,
0xffffffffffff0000, 0x00ffffffffff00ff, 0x00ffffffffffff00, 0xffffffffffffffff,
GGML_TABLE_END()
//#endif
GGML_TABLE_BEGIN(uint64_t, iq2xxs_grid, 256)
0x0808080808080808, 0x080808080808082b, 0x0808080808081919, 0x0808080808082b08,
0x0808080808082b2b, 0x0808080808190819, 0x0808080808191908, 0x08080808082b0808,
0x08080808082b082b, 0x08080808082b2b08, 0x08080808082b2b2b, 0x0808080819080819,
0x0808080819081908, 0x0808080819190808, 0x0808080819192b08, 0x08080808192b0819,
0x08080808192b1908, 0x080808082b080808, 0x080808082b08082b, 0x080808082b082b2b,
0x080808082b2b082b, 0x0808081908080819, 0x0808081908081908, 0x0808081908190808,
0x0808081908191919, 0x0808081919080808, 0x080808192b081908, 0x080808192b192b08,
0x0808082b08080808, 0x0808082b0808082b, 0x0808082b082b082b, 0x0808082b2b08082b,
0x0808190808080819, 0x0808190808081908, 0x0808190808190808, 0x08081908082b0819,
0x08081908082b1908, 0x0808190819080808, 0x080819081908082b, 0x0808190819082b08,
0x08081908192b0808, 0x080819082b080819, 0x080819082b081908, 0x080819082b190808,
0x080819082b2b1908, 0x0808191908080808, 0x080819190808082b, 0x0808191908082b08,
0x08081919082b0808, 0x080819191908192b, 0x08081919192b2b19, 0x080819192b080808,
0x080819192b190819, 0x0808192b08082b19, 0x0808192b08190808, 0x0808192b19080808,
0x0808192b2b081908, 0x0808192b2b2b1908, 0x08082b0808080808, 0x08082b0808081919,
0x08082b0808082b08, 0x08082b0808191908, 0x08082b08082b2b08, 0x08082b0819080819,
0x08082b0819081908, 0x08082b0819190808, 0x08082b081919082b, 0x08082b082b082b08,
0x08082b1908081908, 0x08082b1919080808, 0x08082b2b0808082b, 0x08082b2b08191908,
0x0819080808080819, 0x0819080808081908, 0x0819080808190808, 0x08190808082b0819,
0x0819080819080808, 0x08190808192b0808, 0x081908082b081908, 0x081908082b190808,
0x081908082b191919, 0x0819081908080808, 0x0819081908082b08, 0x08190819082b0808,
0x0819081919190808, 0x0819081919192b2b, 0x081908192b080808, 0x0819082b082b1908,
0x0819082b19081919, 0x0819190808080808, 0x0819190808082b08, 0x08191908082b0808,
0x08191908082b1919, 0x0819190819082b19, 0x081919082b080808, 0x0819191908192b08,
0x08191919192b082b, 0x0819192b08080808, 0x0819192b0819192b, 0x08192b0808080819,
0x08192b0808081908, 0x08192b0808190808, 0x08192b0819080808, 0x08192b082b080819,
0x08192b1908080808, 0x08192b1908081919, 0x08192b192b2b0808, 0x08192b2b19190819,
0x082b080808080808, 0x082b08080808082b, 0x082b080808082b2b, 0x082b080819081908,
0x082b0808192b0819, 0x082b08082b080808, 0x082b08082b08082b, 0x082b0819082b2b19,
0x082b081919082b08, 0x082b082b08080808, 0x082b082b0808082b, 0x082b190808080819,
0x082b190808081908, 0x082b190808190808, 0x082b190819080808, 0x082b19081919192b,
0x082b191908080808, 0x082b191919080819, 0x082b1919192b1908, 0x082b192b2b190808,
0x082b2b0808082b08, 0x082b2b08082b0808, 0x082b2b082b191908, 0x082b2b2b19081908,
0x1908080808080819, 0x1908080808081908, 0x1908080808190808, 0x1908080808192b08,
0x19080808082b0819, 0x19080808082b1908, 0x1908080819080808, 0x1908080819082b08,
0x190808081919192b, 0x19080808192b0808, 0x190808082b080819, 0x190808082b081908,
0x190808082b190808, 0x1908081908080808, 0x19080819082b0808, 0x19080819192b0819,
0x190808192b080808, 0x190808192b081919, 0x1908082b08080819, 0x1908082b08190808,
0x1908082b19082b08, 0x1908082b1919192b, 0x1908082b192b2b08, 0x1908190808080808,
0x1908190808082b08, 0x19081908082b0808, 0x190819082b080808, 0x190819082b192b19,
0x190819190819082b, 0x19081919082b1908, 0x1908192b08080808, 0x19082b0808080819,
0x19082b0808081908, 0x19082b0808190808, 0x19082b0819080808, 0x19082b0819081919,
0x19082b1908080808, 0x19082b1919192b08, 0x19082b19192b0819, 0x19082b192b08082b,
0x19082b2b19081919, 0x19082b2b2b190808, 0x1919080808080808, 0x1919080808082b08,
0x1919080808190819, 0x1919080808192b19, 0x19190808082b0808, 0x191908082b080808,
0x191908082b082b08, 0x1919081908081908, 0x191908191908082b, 0x191908192b2b1908,
0x1919082b2b190819, 0x191919082b190808, 0x191919082b19082b, 0x1919191908082b2b,
0x1919192b08080819, 0x1919192b19191908, 0x19192b0808080808, 0x19192b0808190819,
0x19192b0808192b19, 0x19192b08192b1908, 0x19192b1919080808, 0x19192b2b08082b08,
0x192b080808081908, 0x192b080808190808, 0x192b080819080808, 0x192b0808192b2b08,
0x192b081908080808, 0x192b081919191919, 0x192b082b08192b08, 0x192b082b192b0808,
0x192b190808080808, 0x192b190808081919, 0x192b191908190808, 0x192b19190819082b,
0x192b19192b081908, 0x192b2b081908082b, 0x2b08080808080808, 0x2b0808080808082b,
0x2b08080808082b2b, 0x2b08080819080819, 0x2b0808082b08082b, 0x2b08081908081908,
0x2b08081908192b08, 0x2b08081919080808, 0x2b08082b08190819, 0x2b08190808080819,
0x2b08190808081908, 0x2b08190808190808, 0x2b08190808191919, 0x2b08190819080808,
0x2b081908192b0808, 0x2b08191908080808, 0x2b0819191908192b, 0x2b0819192b191908,
0x2b08192b08082b19, 0x2b08192b19080808, 0x2b08192b192b0808, 0x2b082b080808082b,
0x2b082b1908081908, 0x2b082b2b08190819, 0x2b19080808081908, 0x2b19080808190808,
0x2b190808082b1908, 0x2b19080819080808, 0x2b1908082b2b0819, 0x2b1908190819192b,
0x2b1908192b080808, 0x2b19082b19081919, 0x2b19190808080808, 0x2b191908082b082b,
0x2b19190819081908, 0x2b19191919190819, 0x2b192b082b080819, 0x2b192b19082b0808,
0x2b2b08080808082b, 0x2b2b080819190808, 0x2b2b08082b081919, 0x2b2b081908082b19,
0x2b2b082b08080808, 0x2b2b190808192b08, 0x2b2b2b0819190808, 0x2b2b2b1908081908,
GGML_TABLE_END()
GGML_TABLE_BEGIN(uint64_t, iq2xs_grid, 512)
0x0808080808080808, 0x080808080808082b, 0x0808080808081919, 0x0808080808082b08,
0x0808080808082b2b, 0x0808080808190819, 0x0808080808191908, 0x080808080819192b,
0x0808080808192b19, 0x08080808082b0808, 0x08080808082b082b, 0x08080808082b1919,
0x08080808082b2b08, 0x0808080819080819, 0x0808080819081908, 0x080808081908192b,
0x0808080819082b19, 0x0808080819190808, 0x080808081919082b, 0x0808080819191919,
0x0808080819192b08, 0x08080808192b0819, 0x08080808192b1908, 0x080808082b080808,
0x080808082b08082b, 0x080808082b081919, 0x080808082b082b08, 0x080808082b190819,
0x080808082b191908, 0x080808082b192b19, 0x080808082b2b0808, 0x0808081908080819,
0x0808081908081908, 0x080808190808192b, 0x0808081908082b19, 0x0808081908190808,
0x080808190819082b, 0x0808081908191919, 0x0808081908192b08, 0x0808081908192b2b,
0x08080819082b0819, 0x08080819082b1908, 0x0808081919080808, 0x080808191908082b,
0x0808081919081919, 0x0808081919082b08, 0x0808081919190819, 0x0808081919191908,
0x08080819192b0808, 0x08080819192b2b08, 0x080808192b080819, 0x080808192b081908,
0x080808192b190808, 0x0808082b08080808, 0x0808082b0808082b, 0x0808082b08081919,
0x0808082b08082b08, 0x0808082b08190819, 0x0808082b08191908, 0x0808082b082b0808,
0x0808082b19080819, 0x0808082b19081908, 0x0808082b19190808, 0x0808082b19191919,
0x0808082b2b080808, 0x0808082b2b082b2b, 0x0808190808080819, 0x0808190808081908,
0x080819080808192b, 0x0808190808082b19, 0x0808190808190808, 0x080819080819082b,
0x0808190808191919, 0x0808190808192b08, 0x08081908082b0819, 0x08081908082b1908,
0x0808190819080808, 0x080819081908082b, 0x0808190819081919, 0x0808190819082b08,
0x0808190819190819, 0x0808190819191908, 0x080819081919192b, 0x08081908192b0808,
0x080819082b080819, 0x080819082b081908, 0x080819082b190808, 0x0808191908080808,
0x080819190808082b, 0x0808191908081919, 0x0808191908082b08, 0x0808191908190819,
0x0808191908191908, 0x08081919082b0808, 0x0808191919080819, 0x0808191919081908,
0x0808191919190808, 0x08081919192b0819, 0x080819192b080808, 0x0808192b08080819,
0x0808192b08081908, 0x0808192b08190808, 0x0808192b082b192b, 0x0808192b19080808,
0x0808192b1908082b, 0x0808192b2b081908, 0x08082b0808080808, 0x08082b080808082b,
0x08082b0808081919, 0x08082b0808082b08, 0x08082b0808082b2b, 0x08082b0808190819,
0x08082b0808191908, 0x08082b08082b0808, 0x08082b08082b1919, 0x08082b0819080819,
0x08082b0819081908, 0x08082b0819190808, 0x08082b0819192b08, 0x08082b082b080808,
0x08082b082b2b0808, 0x08082b082b2b2b2b, 0x08082b1908080819, 0x08082b1908081908,
0x08082b1908190808, 0x08082b1919080808, 0x08082b192b080819, 0x08082b192b082b19,
0x08082b2b08080808, 0x08082b2b082b0808, 0x08082b2b082b2b08, 0x08082b2b2b19192b,
0x08082b2b2b2b0808, 0x0819080808080819, 0x0819080808081908, 0x081908080808192b,
0x0819080808082b19, 0x0819080808190808, 0x081908080819082b, 0x0819080808191919,
0x0819080808192b08, 0x08190808082b0819, 0x08190808082b1908, 0x0819080819080808,
0x081908081908082b, 0x0819080819081919, 0x0819080819082b08, 0x0819080819190819,
0x0819080819191908, 0x08190808192b0808, 0x08190808192b2b2b, 0x081908082b080819,
0x081908082b081908, 0x081908082b190808, 0x0819081908080808, 0x081908190808082b,
0x0819081908081919, 0x0819081908082b08, 0x0819081908190819, 0x0819081908191908,
0x08190819082b0808, 0x0819081919080819, 0x0819081919081908, 0x0819081919190808,
0x081908192b080808, 0x081908192b191908, 0x081908192b19192b, 0x0819082b08080819,
0x0819082b08081908, 0x0819082b0808192b, 0x0819082b08190808, 0x0819082b19080808,
0x0819082b192b0808, 0x0819190808080808, 0x081919080808082b, 0x0819190808081919,
0x0819190808082b08, 0x0819190808190819, 0x0819190808191908, 0x08191908082b0808,
0x0819190819080819, 0x0819190819081908, 0x0819190819082b19, 0x0819190819190808,
0x08191908192b1908, 0x081919082b080808, 0x0819191908080819, 0x0819191908081908,
0x0819191908190808, 0x0819191919080808, 0x0819192b08080808, 0x0819192b08191908,
0x0819192b19082b19, 0x08192b0808080819, 0x08192b0808081908, 0x08192b0808190808,
0x08192b080819082b, 0x08192b0819080808, 0x08192b0819191908, 0x08192b082b08192b,
0x08192b1908080808, 0x08192b1908081919, 0x08192b19192b192b, 0x08192b2b19190819,
0x08192b2b2b2b2b19, 0x082b080808080808, 0x082b08080808082b, 0x082b080808081919,
0x082b080808082b08, 0x082b080808082b2b, 0x082b080808190819, 0x082b080808191908,
0x082b0808082b0808, 0x082b080819080819, 0x082b080819081908, 0x082b080819190808,
0x082b08082b080808, 0x082b08082b2b0808, 0x082b081908080819, 0x082b081908081908,
0x082b081908190808, 0x082b081919080808, 0x082b081919082b08, 0x082b0819192b1919,
0x082b082b08080808, 0x082b082b082b082b, 0x082b082b2b080808, 0x082b082b2b2b2b08,
0x082b190808080819, 0x082b190808081908, 0x082b190808190808, 0x082b1908082b2b19,
0x082b190819080808, 0x082b191908080808, 0x082b191919080819, 0x082b19191919082b,
0x082b19192b192b19, 0x082b192b08080819, 0x082b192b08192b2b, 0x082b192b2b2b192b,
0x082b2b0808080808, 0x082b2b0808082b08, 0x082b2b0808082b2b, 0x082b2b08082b0808,
0x082b2b0819191919, 0x082b2b082b082b08, 0x082b2b082b2b082b, 0x082b2b19192b2b08,
0x082b2b192b190808, 0x082b2b2b08082b08, 0x082b2b2b082b0808, 0x082b2b2b2b08082b,
0x082b2b2b2b082b08, 0x082b2b2b2b082b2b, 0x1908080808080819, 0x1908080808081908,
0x190808080808192b, 0x1908080808082b19, 0x1908080808190808, 0x190808080819082b,
0x1908080808191919, 0x1908080808192b08, 0x19080808082b0819, 0x19080808082b1908,
0x1908080819080808, 0x190808081908082b, 0x1908080819081919, 0x1908080819082b08,
0x1908080819082b2b, 0x1908080819190819, 0x1908080819191908, 0x19080808192b0808,
0x19080808192b1919, 0x190808082b080819, 0x190808082b081908, 0x190808082b190808,
0x1908081908080808, 0x190808190808082b, 0x1908081908081919, 0x1908081908082b08,
0x1908081908190819, 0x1908081908191908, 0x19080819082b0808, 0x1908081919080819,
0x1908081919081908, 0x1908081919190808, 0x190808192b080808, 0x190808192b081919,
0x190808192b2b082b, 0x1908082b08080819, 0x1908082b08081908, 0x1908082b08190808,
0x1908082b0819082b, 0x1908082b082b2b19, 0x1908082b19080808, 0x1908190808080808,
0x190819080808082b, 0x1908190808081919, 0x1908190808082b08, 0x1908190808190819,
0x1908190808191908, 0x1908190808192b19, 0x19081908082b0808, 0x1908190819080819,
0x1908190819081908, 0x1908190819190808, 0x190819082b080808, 0x190819082b191908,
0x1908191908080819, 0x1908191908081908, 0x1908191908190808, 0x19081919082b1908,
0x1908191919080808, 0x190819192b192b2b, 0x1908192b08080808, 0x1908192b08082b2b,
0x1908192b19081908, 0x1908192b19190808, 0x19082b0808080819, 0x19082b0808081908,
0x19082b0808190808, 0x19082b0819080808, 0x19082b0819081919, 0x19082b0819191908,
0x19082b08192b082b, 0x19082b1908080808, 0x19082b1908190819, 0x19082b1919081908,
0x19082b1919190808, 0x19082b19192b2b19, 0x19082b2b08081908, 0x1919080808080808,
0x191908080808082b, 0x1919080808081919, 0x1919080808082b08, 0x1919080808190819,
0x1919080808191908, 0x19190808082b0808, 0x19190808082b2b08, 0x1919080819080819,
0x1919080819081908, 0x1919080819190808, 0x191908082b080808, 0x1919081908080819,
0x1919081908081908, 0x1919081908190808, 0x1919081908191919, 0x1919081919080808,
0x191908191908082b, 0x1919082b08080808, 0x1919082b19081908, 0x1919082b2b2b2b2b,
0x1919190808080819, 0x1919190808081908, 0x1919190808190808, 0x19191908082b0819,
0x1919190819080808, 0x19191908192b0808, 0x191919082b080819, 0x191919082b2b0819,
0x1919191908080808, 0x1919191908082b08, 0x191919192b080808, 0x191919192b082b08,
0x1919192b082b0819, 0x1919192b192b2b08, 0x1919192b2b2b0819, 0x19192b0808080808,
0x19192b0808191908, 0x19192b0819080819, 0x19192b0819190808, 0x19192b082b192b19,
0x19192b1908192b2b, 0x19192b1919080808, 0x19192b191908082b, 0x19192b2b2b081919,
0x192b080808080819, 0x192b080808081908, 0x192b080808190808, 0x192b080819080808,
0x192b080819191908, 0x192b0808192b082b, 0x192b08082b08192b, 0x192b08082b2b2b19,
0x192b081908080808, 0x192b082b082b1908, 0x192b082b19082b2b, 0x192b082b2b19082b,
0x192b190808080808, 0x192b19080819192b, 0x192b191908190808, 0x192b191919080808,
0x192b191919081919, 0x192b19192b2b1908, 0x192b2b0808080819, 0x192b2b08192b2b2b,
0x192b2b19082b1919, 0x192b2b2b0808192b, 0x192b2b2b19191908, 0x192b2b2b192b082b,
0x2b08080808080808, 0x2b0808080808082b, 0x2b08080808081919, 0x2b08080808082b08,
0x2b08080808190819, 0x2b08080808191908, 0x2b080808082b0808, 0x2b080808082b2b2b,
0x2b08080819080819, 0x2b08080819081908, 0x2b08080819190808, 0x2b0808082b080808,
0x2b0808082b08082b, 0x2b0808082b2b2b08, 0x2b0808082b2b2b2b, 0x2b08081908080819,
0x2b08081908081908, 0x2b0808190808192b, 0x2b08081908190808, 0x2b08081919080808,
0x2b08081919190819, 0x2b08081919192b19, 0x2b08082b08080808, 0x2b08082b082b0808,
0x2b08082b2b080808, 0x2b08082b2b08082b, 0x2b08082b2b2b0808, 0x2b08082b2b2b2b08,
0x2b08190808080819, 0x2b08190808081908, 0x2b08190808190808, 0x2b0819080819082b,
0x2b08190808191919, 0x2b08190819080808, 0x2b081908192b0808, 0x2b0819082b082b19,
0x2b08191908080808, 0x2b08191919081908, 0x2b0819192b2b1919, 0x2b08192b08192b08,
0x2b08192b192b2b2b, 0x2b082b0808080808, 0x2b082b0808082b08, 0x2b082b08082b1919,
0x2b082b0819192b2b, 0x2b082b082b080808, 0x2b082b082b08082b, 0x2b082b082b2b2b08,
0x2b082b190808192b, 0x2b082b2b082b082b, 0x2b082b2b2b080808, 0x2b082b2b2b082b08,
0x2b082b2b2b19192b, 0x2b082b2b2b2b2b08, 0x2b19080808080819, 0x2b19080808081908,
0x2b19080808190808, 0x2b19080819080808, 0x2b1908081919192b, 0x2b1908082b081908,
0x2b19081908080808, 0x2b190819082b082b, 0x2b190819192b1908, 0x2b19082b1919192b,
0x2b19082b2b082b19, 0x2b19190808080808, 0x2b19190808081919, 0x2b19190819081908,
0x2b19190819190808, 0x2b19190819192b08, 0x2b191919082b2b19, 0x2b1919192b190808,
0x2b1919192b19082b, 0x2b19192b19080819, 0x2b192b0819190819, 0x2b192b082b2b192b,
0x2b192b1919082b19, 0x2b192b2b08191919, 0x2b192b2b192b0808, 0x2b2b080808080808,
0x2b2b08080808082b, 0x2b2b080808082b08, 0x2b2b080808082b2b, 0x2b2b0808082b0808,
0x2b2b0808082b2b2b, 0x2b2b08082b2b0808, 0x2b2b081919190819, 0x2b2b081919192b19,
0x2b2b08192b2b192b, 0x2b2b082b08080808, 0x2b2b082b0808082b, 0x2b2b082b08082b08,
0x2b2b082b082b2b2b, 0x2b2b082b2b080808, 0x2b2b082b2b2b0808, 0x2b2b190819080808,
0x2b2b19082b191919, 0x2b2b192b192b1919, 0x2b2b192b2b192b08, 0x2b2b2b0808082b2b,
0x2b2b2b08082b0808, 0x2b2b2b08082b082b, 0x2b2b2b08082b2b08, 0x2b2b2b082b2b0808,
0x2b2b2b082b2b2b08, 0x2b2b2b1908081908, 0x2b2b2b192b081908, 0x2b2b2b192b08192b,
0x2b2b2b2b082b2b08, 0x2b2b2b2b082b2b2b, 0x2b2b2b2b2b190819, 0x2b2b2b2b2b2b2b2b,
GGML_TABLE_END()
GGML_TABLE_BEGIN(uint64_t, iq2s_grid, 1024)
0x0808080808080808, 0x080808080808082b, 0x0808080808081919, 0x0808080808082b08,
0x0808080808082b2b, 0x0808080808190819, 0x0808080808191908, 0x080808080819192b,
0x0808080808192b19, 0x08080808082b0808, 0x08080808082b082b, 0x08080808082b1919,
0x08080808082b2b08, 0x0808080819080819, 0x0808080819081908, 0x080808081908192b,
0x0808080819082b19, 0x0808080819190808, 0x080808081919082b, 0x0808080819191919,
0x0808080819192b08, 0x08080808192b0819, 0x08080808192b1908, 0x08080808192b192b,
0x08080808192b2b19, 0x080808082b080808, 0x080808082b08082b, 0x080808082b081919,
0x080808082b082b08, 0x080808082b190819, 0x080808082b191908, 0x080808082b2b0808,
0x080808082b2b1919, 0x080808082b2b2b2b, 0x0808081908080819, 0x0808081908081908,
0x080808190808192b, 0x0808081908082b19, 0x0808081908190808, 0x080808190819082b,
0x0808081908191919, 0x0808081908192b08, 0x08080819082b0819, 0x08080819082b1908,
0x0808081919080808, 0x080808191908082b, 0x0808081919081919, 0x0808081919082b08,
0x0808081919190819, 0x0808081919191908, 0x080808191919192b, 0x0808081919192b19,
0x08080819192b0808, 0x08080819192b1919, 0x08080819192b2b08, 0x080808192b080819,
0x080808192b081908, 0x080808192b190808, 0x080808192b19082b, 0x080808192b191919,
0x080808192b2b0819, 0x080808192b2b1908, 0x0808082b08080808, 0x0808082b0808082b,
0x0808082b08081919, 0x0808082b08082b08, 0x0808082b08190819, 0x0808082b08191908,
0x0808082b082b0808, 0x0808082b082b2b2b, 0x0808082b19080819, 0x0808082b19081908,
0x0808082b1908192b, 0x0808082b19082b19, 0x0808082b19190808, 0x0808082b19191919,
0x0808082b2b080808, 0x0808082b2b081919, 0x0808082b2b082b2b, 0x0808082b2b191908,
0x0808082b2b2b082b, 0x0808190808080819, 0x0808190808081908, 0x080819080808192b,
0x0808190808082b19, 0x0808190808190808, 0x080819080819082b, 0x0808190808191919,
0x0808190808192b08, 0x08081908082b0819, 0x08081908082b1908, 0x08081908082b192b,
0x08081908082b2b19, 0x0808190819080808, 0x080819081908082b, 0x0808190819081919,
0x0808190819082b08, 0x0808190819082b2b, 0x0808190819190819, 0x0808190819191908,
0x080819081919192b, 0x0808190819192b19, 0x08081908192b0808, 0x08081908192b082b,
0x08081908192b1919, 0x080819082b080819, 0x080819082b081908, 0x080819082b08192b,
0x080819082b082b19, 0x080819082b190808, 0x080819082b191919, 0x080819082b192b08,
0x080819082b2b0819, 0x080819082b2b1908, 0x0808191908080808, 0x080819190808082b,
0x0808191908081919, 0x0808191908082b08, 0x0808191908082b2b, 0x0808191908190819,
0x0808191908191908, 0x080819190819192b, 0x0808191908192b19, 0x08081919082b0808,
0x08081919082b1919, 0x08081919082b2b08, 0x0808191919080819, 0x0808191919081908,
0x080819191908192b, 0x0808191919082b19, 0x0808191919190808, 0x080819191919082b,
0x0808191919191919, 0x0808191919192b08, 0x08081919192b0819, 0x08081919192b1908,
0x080819192b080808, 0x080819192b08082b, 0x080819192b081919, 0x080819192b082b08,
0x080819192b190819, 0x080819192b191908, 0x080819192b2b0808, 0x0808192b08080819,
0x0808192b08081908, 0x0808192b0808192b, 0x0808192b08082b19, 0x0808192b08190808,
0x0808192b08191919, 0x0808192b19080808, 0x0808192b19081919, 0x0808192b19082b08,
0x0808192b19190819, 0x0808192b19191908, 0x0808192b192b0808, 0x0808192b2b080819,
0x0808192b2b081908, 0x0808192b2b190808, 0x08082b0808080808, 0x08082b080808082b,
0x08082b0808081919, 0x08082b0808082b08, 0x08082b0808190819, 0x08082b0808191908,
0x08082b080819192b, 0x08082b0808192b19, 0x08082b08082b0808, 0x08082b08082b1919,
0x08082b08082b2b2b, 0x08082b0819080819, 0x08082b0819081908, 0x08082b081908192b,
0x08082b0819082b19, 0x08082b0819190808, 0x08082b081919082b, 0x08082b0819191919,
0x08082b0819192b08, 0x08082b08192b0819, 0x08082b08192b1908, 0x08082b082b080808,
0x08082b082b081919, 0x08082b082b191908, 0x08082b082b2b2b2b, 0x08082b1908080819,
0x08082b1908081908, 0x08082b1908190808, 0x08082b190819082b, 0x08082b1908191919,
0x08082b1908192b08, 0x08082b19082b0819, 0x08082b1919080808, 0x08082b1919081919,
0x08082b1919082b08, 0x08082b1919190819, 0x08082b1919191908, 0x08082b19192b0808,
0x08082b192b080819, 0x08082b192b190808, 0x08082b2b08080808, 0x08082b2b08190819,
0x08082b2b08191908, 0x08082b2b082b082b, 0x08082b2b082b2b08, 0x08082b2b082b2b2b,
0x08082b2b19190808, 0x08082b2b2b192b19, 0x0819080808080819, 0x0819080808081908,
0x081908080808192b, 0x0819080808082b19, 0x0819080808190808, 0x081908080819082b,
0x0819080808191919, 0x0819080808192b08, 0x08190808082b0819, 0x08190808082b1908,
0x08190808082b192b, 0x0819080819080808, 0x081908081908082b, 0x0819080819081919,
0x0819080819082b08, 0x0819080819190819, 0x0819080819191908, 0x081908081919192b,
0x0819080819192b19, 0x08190808192b0808, 0x08190808192b082b, 0x08190808192b1919,
0x08190808192b2b08, 0x081908082b080819, 0x081908082b081908, 0x081908082b08192b,
0x081908082b190808, 0x081908082b191919, 0x081908082b192b08, 0x081908082b2b0819,
0x081908082b2b1908, 0x0819081908080808, 0x081908190808082b, 0x0819081908081919,
0x0819081908082b08, 0x0819081908082b2b, 0x0819081908190819, 0x0819081908191908,
0x081908190819192b, 0x0819081908192b19, 0x08190819082b0808, 0x08190819082b082b,
0x08190819082b1919, 0x08190819082b2b08, 0x0819081919080819, 0x0819081919081908,
0x081908191908192b, 0x0819081919082b19, 0x0819081919190808, 0x081908191919082b,
0x0819081919191919, 0x0819081919192b08, 0x08190819192b0819, 0x08190819192b1908,
0x081908192b080808, 0x081908192b08082b, 0x081908192b081919, 0x081908192b082b08,
0x081908192b190819, 0x081908192b191908, 0x0819082b08080819, 0x0819082b08081908,
0x0819082b08082b19, 0x0819082b08190808, 0x0819082b08191919, 0x0819082b082b0819,
0x0819082b082b1908, 0x0819082b19080808, 0x0819082b19081919, 0x0819082b19190819,
0x0819082b19191908, 0x0819082b2b080819, 0x0819082b2b081908, 0x0819082b2b190808,
0x0819190808080808, 0x081919080808082b, 0x0819190808081919, 0x0819190808082b08,
0x0819190808190819, 0x0819190808191908, 0x081919080819192b, 0x0819190808192b19,
0x08191908082b0808, 0x08191908082b1919, 0x08191908082b2b08, 0x0819190819080819,
0x0819190819081908, 0x081919081908192b, 0x0819190819082b19, 0x0819190819190808,
0x081919081919082b, 0x0819190819191919, 0x0819190819192b08, 0x08191908192b0819,
0x08191908192b1908, 0x081919082b080808, 0x081919082b08082b, 0x081919082b081919,
0x081919082b082b08, 0x081919082b190819, 0x081919082b191908, 0x081919082b2b0808,
0x0819191908080819, 0x0819191908081908, 0x081919190808192b, 0x0819191908082b19,
0x0819191908190808, 0x081919190819082b, 0x0819191908191919, 0x0819191908192b08,
0x08191919082b0819, 0x08191919082b1908, 0x0819191919080808, 0x081919191908082b,
0x0819191919081919, 0x0819191919082b08, 0x0819191919190819, 0x0819191919191908,
0x08191919192b0808, 0x081919192b080819, 0x081919192b081908, 0x081919192b190808,
0x0819192b08080808, 0x0819192b08081919, 0x0819192b08082b08, 0x0819192b08190819,
0x0819192b08191908, 0x0819192b082b0808, 0x0819192b19080819, 0x0819192b19081908,
0x0819192b19190808, 0x0819192b2b080808, 0x0819192b2b2b2b2b, 0x08192b0808080819,
0x08192b0808081908, 0x08192b080808192b, 0x08192b0808082b19, 0x08192b0808190808,
0x08192b0808191919, 0x08192b0808192b08, 0x08192b08082b0819, 0x08192b0819080808,
0x08192b081908082b, 0x08192b0819081919, 0x08192b0819082b08, 0x08192b0819190819,
0x08192b0819191908, 0x08192b08192b0808, 0x08192b082b080819, 0x08192b082b081908,
0x08192b1908080808, 0x08192b190808082b, 0x08192b1908081919, 0x08192b1908082b08,
0x08192b1908190819, 0x08192b1908191908, 0x08192b19082b0808, 0x08192b1919080819,
0x08192b1919081908, 0x08192b1919190808, 0x08192b19192b2b19, 0x08192b192b2b082b,
0x08192b2b08081908, 0x08192b2b08190808, 0x08192b2b19080808, 0x08192b2b1919192b,
0x082b080808080808, 0x082b08080808082b, 0x082b080808081919, 0x082b080808082b08,
0x082b080808190819, 0x082b080808191908, 0x082b08080819192b, 0x082b080808192b19,
0x082b0808082b0808, 0x082b0808082b1919, 0x082b0808082b2b2b, 0x082b080819080819,
0x082b080819081908, 0x082b080819190808, 0x082b08081919082b, 0x082b080819191919,
0x082b0808192b1908, 0x082b08082b080808, 0x082b08082b082b2b, 0x082b08082b191908,
0x082b08082b2b2b2b, 0x082b081908080819, 0x082b081908081908, 0x082b081908190808,
0x082b08190819082b, 0x082b081908191919, 0x082b0819082b0819, 0x082b081919080808,
0x082b08191908082b, 0x082b081919081919, 0x082b081919190819, 0x082b081919191908,
0x082b0819192b0808, 0x082b08192b080819, 0x082b08192b081908, 0x082b08192b190808,
0x082b082b08080808, 0x082b082b08082b2b, 0x082b082b082b082b, 0x082b082b082b2b08,
0x082b082b082b2b2b, 0x082b082b19081908, 0x082b082b19190808, 0x082b082b2b082b08,
0x082b082b2b082b2b, 0x082b082b2b2b2b08, 0x082b190808080819, 0x082b190808081908,
0x082b19080808192b, 0x082b190808082b19, 0x082b190808190808, 0x082b190808191919,
0x082b190808192b08, 0x082b1908082b0819, 0x082b1908082b1908, 0x082b190819080808,
0x082b19081908082b, 0x082b190819081919, 0x082b190819082b08, 0x082b190819190819,
0x082b190819191908, 0x082b1908192b0808, 0x082b19082b080819, 0x082b19082b081908,
0x082b19082b190808, 0x082b191908080808, 0x082b191908081919, 0x082b191908082b08,
0x082b191908190819, 0x082b191908191908, 0x082b1919082b0808, 0x082b191919080819,
0x082b191919081908, 0x082b191919190808, 0x082b1919192b192b, 0x082b19192b080808,
0x082b192b08080819, 0x082b192b08081908, 0x082b192b08190808, 0x082b192b19080808,
0x082b192b19192b19, 0x082b2b0808080808, 0x082b2b0808081919, 0x082b2b0808190819,
0x082b2b0808191908, 0x082b2b0819080819, 0x082b2b0819081908, 0x082b2b0819190808,
0x082b2b082b082b2b, 0x082b2b082b2b2b2b, 0x082b2b1908080819, 0x082b2b1908081908,
0x082b2b1908190808, 0x082b2b192b191919, 0x082b2b2b08082b2b, 0x082b2b2b082b082b,
0x082b2b2b192b1908, 0x082b2b2b2b082b08, 0x082b2b2b2b082b2b, 0x1908080808080819,
0x1908080808081908, 0x190808080808192b, 0x1908080808082b19, 0x1908080808190808,
0x190808080819082b, 0x1908080808191919, 0x1908080808192b08, 0x1908080808192b2b,
0x19080808082b0819, 0x19080808082b1908, 0x19080808082b192b, 0x1908080819080808,
0x190808081908082b, 0x1908080819081919, 0x1908080819082b08, 0x1908080819082b2b,
0x1908080819190819, 0x1908080819191908, 0x190808081919192b, 0x1908080819192b19,
0x19080808192b0808, 0x19080808192b082b, 0x19080808192b1919, 0x190808082b080819,
0x190808082b081908, 0x190808082b190808, 0x190808082b191919, 0x190808082b192b08,
0x190808082b2b0819, 0x190808082b2b1908, 0x1908081908080808, 0x190808190808082b,
0x1908081908081919, 0x1908081908082b08, 0x1908081908190819, 0x1908081908191908,
0x190808190819192b, 0x1908081908192b19, 0x19080819082b0808, 0x19080819082b082b,
0x19080819082b1919, 0x1908081919080819, 0x1908081919081908, 0x190808191908192b,
0x1908081919082b19, 0x1908081919190808, 0x190808191919082b, 0x1908081919191919,
0x1908081919192b08, 0x19080819192b0819, 0x19080819192b1908, 0x190808192b080808,
0x190808192b08082b, 0x190808192b081919, 0x190808192b082b08, 0x190808192b190819,
0x190808192b191908, 0x190808192b2b0808, 0x1908082b08080819, 0x1908082b08081908,
0x1908082b08190808, 0x1908082b0819082b, 0x1908082b08191919, 0x1908082b08192b08,
0x1908082b082b1908, 0x1908082b19080808, 0x1908082b19081919, 0x1908082b19082b08,
0x1908082b19190819, 0x1908082b19191908, 0x1908082b192b0808, 0x1908082b2b080819,
0x1908082b2b081908, 0x1908190808080808, 0x190819080808082b, 0x1908190808081919,
0x1908190808082b08, 0x1908190808082b2b, 0x1908190808190819, 0x1908190808191908,
0x190819080819192b, 0x1908190808192b19, 0x19081908082b0808, 0x19081908082b082b,
0x19081908082b1919, 0x19081908082b2b08, 0x1908190819080819, 0x1908190819081908,
0x190819081908192b, 0x1908190819082b19, 0x1908190819190808, 0x190819081919082b,
0x1908190819191919, 0x1908190819192b08, 0x19081908192b0819, 0x19081908192b1908,
0x190819082b080808, 0x190819082b08082b, 0x190819082b081919, 0x190819082b082b08,
0x190819082b190819, 0x190819082b191908, 0x190819082b2b0808, 0x1908191908080819,
0x1908191908081908, 0x190819190808192b, 0x1908191908082b19, 0x1908191908190808,
0x190819190819082b, 0x1908191908191919, 0x1908191908192b08, 0x19081919082b0819,
0x19081919082b1908, 0x1908191919080808, 0x190819191908082b, 0x1908191919081919,
0x1908191919082b08, 0x1908191919190819, 0x1908191919191908, 0x19081919192b0808,
0x19081919192b2b2b, 0x190819192b080819, 0x190819192b081908, 0x190819192b190808,
0x1908192b08080808, 0x1908192b0808082b, 0x1908192b08081919, 0x1908192b08082b08,
0x1908192b08190819, 0x1908192b08191908, 0x1908192b082b0808, 0x1908192b19080819,
0x1908192b19081908, 0x1908192b19190808, 0x1908192b2b080808, 0x1908192b2b2b1919,
0x19082b0808080819, 0x19082b0808081908, 0x19082b0808082b19, 0x19082b0808190808,
0x19082b080819082b, 0x19082b0808191919, 0x19082b0808192b08, 0x19082b08082b0819,
0x19082b08082b1908, 0x19082b0819080808, 0x19082b081908082b, 0x19082b0819081919,
0x19082b0819082b08, 0x19082b0819190819, 0x19082b0819191908, 0x19082b08192b0808,
0x19082b082b081908, 0x19082b082b190808, 0x19082b1908080808, 0x19082b190808082b,
0x19082b1908081919, 0x19082b1908082b08, 0x19082b1908190819, 0x19082b1908191908,
0x19082b19082b0808, 0x19082b1919080819, 0x19082b1919081908, 0x19082b1919190808,
0x19082b192b080808, 0x19082b192b19192b, 0x19082b2b08080819, 0x19082b2b08081908,
0x19082b2b08190808, 0x19082b2b19080808, 0x1919080808080808, 0x191908080808082b,
0x1919080808081919, 0x1919080808082b08, 0x1919080808190819, 0x1919080808191908,
0x191908080819192b, 0x1919080808192b19, 0x19190808082b0808, 0x19190808082b082b,
0x19190808082b1919, 0x19190808082b2b08, 0x1919080819080819, 0x1919080819081908,
0x191908081908192b, 0x1919080819082b19, 0x1919080819190808, 0x191908081919082b,
0x1919080819191919, 0x1919080819192b08, 0x19190808192b0819, 0x19190808192b1908,
0x191908082b080808, 0x191908082b08082b, 0x191908082b081919, 0x191908082b082b08,
0x191908082b190819, 0x191908082b191908, 0x1919081908080819, 0x1919081908081908,
0x191908190808192b, 0x1919081908082b19, 0x1919081908190808, 0x191908190819082b,
0x1919081908191919, 0x1919081908192b08, 0x19190819082b0819, 0x19190819082b1908,
0x1919081919080808, 0x191908191908082b, 0x1919081919081919, 0x1919081919082b08,
0x1919081919190819, 0x1919081919191908, 0x19190819192b0808, 0x191908192b080819,
0x191908192b081908, 0x191908192b190808, 0x1919082b08080808, 0x1919082b08081919,
0x1919082b08082b08, 0x1919082b08190819, 0x1919082b08191908, 0x1919082b082b0808,
0x1919082b19080819, 0x1919082b19081908, 0x1919082b19190808, 0x1919082b192b2b19,
0x1919082b2b080808, 0x1919190808080819, 0x1919190808081908, 0x191919080808192b,
0x1919190808082b19, 0x1919190808190808, 0x191919080819082b, 0x1919190808191919,
0x1919190808192b08, 0x19191908082b0819, 0x19191908082b1908, 0x1919190819080808,
0x191919081908082b, 0x1919190819081919, 0x1919190819082b08, 0x1919190819190819,
0x1919190819191908, 0x19191908192b0808, 0x191919082b080819, 0x191919082b081908,
0x191919082b190808, 0x1919191908080808, 0x191919190808082b, 0x1919191908081919,
0x1919191908082b08, 0x1919191908190819, 0x1919191908191908, 0x19191919082b0808,
0x1919191919080819, 0x1919191919081908, 0x1919191919190808, 0x191919192b080808,
0x1919192b08080819, 0x1919192b08081908, 0x1919192b08190808, 0x1919192b082b192b,
0x1919192b19080808, 0x19192b0808080808, 0x19192b080808082b, 0x19192b0808081919,
0x19192b0808082b08, 0x19192b0808190819, 0x19192b0808191908, 0x19192b08082b0808,
0x19192b0819080819, 0x19192b0819081908, 0x19192b0819190808, 0x19192b0819192b2b,
0x19192b082b080808, 0x19192b1908080819, 0x19192b1908081908, 0x19192b1908190808,
0x19192b1919080808, 0x19192b2b08080808, 0x19192b2b08192b19, 0x19192b2b2b081919,
0x19192b2b2b2b2b08, 0x192b080808080819, 0x192b080808081908, 0x192b08080808192b,
0x192b080808190808, 0x192b08080819082b, 0x192b080808191919, 0x192b080808192b08,
0x192b0808082b0819, 0x192b0808082b1908, 0x192b080819080808, 0x192b080819081919,
0x192b080819082b08, 0x192b080819190819, 0x192b080819191908, 0x192b0808192b0808,
0x192b08082b081908, 0x192b08082b190808, 0x192b081908080808, 0x192b08190808082b,
0x192b081908081919, 0x192b081908082b08, 0x192b081908190819, 0x192b081908191908,
0x192b0819082b0808, 0x192b081919080819, 0x192b081919081908, 0x192b081919190808,
0x192b08192b080808, 0x192b08192b192b19, 0x192b082b08081908, 0x192b082b08190808,
0x192b082b19080808, 0x192b082b1919192b, 0x192b082b2b2b0819, 0x192b190808080808,
0x192b190808081919, 0x192b190808082b08, 0x192b190808190819, 0x192b190808191908,
0x192b1908082b0808, 0x192b190819080819, 0x192b190819081908, 0x192b190819190808,
0x192b19082b080808, 0x192b191908080819, 0x192b191908081908, 0x192b191908190808,
0x192b191919080808, 0x192b191919082b2b, 0x192b1919192b2b08, 0x192b19192b19082b,
0x192b192b08080808, 0x192b192b2b191908, 0x192b2b0808080819, 0x192b2b0808081908,
0x192b2b0808190808, 0x192b2b08192b1919, 0x192b2b082b192b08, 0x192b2b1908080808,
0x192b2b19082b2b2b, 0x192b2b2b1908082b, 0x192b2b2b2b2b0819, 0x2b08080808080808,
0x2b0808080808082b, 0x2b08080808081919, 0x2b08080808082b08, 0x2b08080808190819,
0x2b08080808191908, 0x2b08080808192b19, 0x2b080808082b0808, 0x2b080808082b1919,
0x2b08080819080819, 0x2b08080819081908, 0x2b08080819190808, 0x2b0808081919082b,
0x2b08080819191919, 0x2b08080819192b08, 0x2b080808192b0819, 0x2b0808082b080808,
0x2b0808082b081919, 0x2b0808082b190819, 0x2b0808082b191908, 0x2b08081908080819,
0x2b08081908081908, 0x2b08081908082b19, 0x2b08081908190808, 0x2b0808190819082b,
0x2b08081908191919, 0x2b08081908192b08, 0x2b080819082b0819, 0x2b080819082b1908,
0x2b08081919080808, 0x2b0808191908082b, 0x2b08081919081919, 0x2b08081919082b08,
0x2b08081919190819, 0x2b08081919191908, 0x2b0808192b080819, 0x2b0808192b081908,
0x2b0808192b190808, 0x2b0808192b2b2b19, 0x2b08082b08080808, 0x2b08082b08081919,
0x2b08082b08082b2b, 0x2b08082b08190819, 0x2b08082b08191908, 0x2b08082b19080819,
0x2b08082b19081908, 0x2b08082b19190808, 0x2b08190808080819, 0x2b08190808081908,
0x2b0819080808192b, 0x2b08190808082b19, 0x2b08190808190808, 0x2b0819080819082b,
0x2b08190808191919, 0x2b08190808192b08, 0x2b081908082b0819, 0x2b08190819080808,
0x2b0819081908082b, 0x2b08190819081919, 0x2b08190819082b08, 0x2b08190819190819,
0x2b08190819191908, 0x2b081908192b0808, 0x2b0819082b080819, 0x2b0819082b081908,
0x2b0819082b190808, 0x2b08191908080808, 0x2b0819190808082b, 0x2b08191908081919,
0x2b08191908082b08, 0x2b08191908190819, 0x2b08191908191908, 0x2b081919082b0808,
0x2b08191919080819, 0x2b08191919081908, 0x2b08191919190808, 0x2b0819192b080808,
0x2b0819192b082b2b, 0x2b08192b08080819, 0x2b08192b08081908, 0x2b08192b08190808,
0x2b08192b082b2b19, 0x2b08192b19080808, 0x2b082b0808080808, 0x2b082b0808081919,
0x2b082b0808190819, 0x2b082b0808191908, 0x2b082b0819080819, 0x2b082b0819081908,
0x2b082b0819190808, 0x2b082b082b2b082b, 0x2b082b1908080819, 0x2b082b1908081908,
0x2b082b1919080808, 0x2b082b19192b1919, 0x2b082b2b082b082b, 0x2b082b2b19192b08,
0x2b082b2b19192b2b, 0x2b082b2b2b08082b, 0x2b082b2b2b2b082b, 0x2b19080808080819,
0x2b19080808081908, 0x2b19080808082b19, 0x2b19080808190808, 0x2b1908080819082b,
0x2b19080808191919, 0x2b19080808192b08, 0x2b190808082b1908, 0x2b19080819080808,
0x2b1908081908082b, 0x2b19080819081919, 0x2b19080819082b08, 0x2b19080819190819,
0x2b19080819191908, 0x2b190808192b0808, 0x2b1908082b080819, 0x2b1908082b081908,
0x2b1908082b190808, 0x2b19081908080808, 0x2b19081908081919, 0x2b19081908190819,
0x2b19081908191908, 0x2b19081919080819, 0x2b19081919081908, 0x2b19081919190808,
0x2b19081919192b2b, 0x2b19082b08080819, 0x2b19082b08081908, 0x2b19082b08190808,
0x2b19082b19080808, 0x2b19082b2b2b192b, 0x2b19190808080808, 0x2b1919080808082b,
0x2b19190808081919, 0x2b19190808082b08, 0x2b19190808190819, 0x2b19190808191908,
0x2b191908082b0808, 0x2b19190819080819, 0x2b19190819081908, 0x2b19190819190808,
0x2b1919082b080808, 0x2b1919082b19192b, 0x2b19191908080819, 0x2b19191908081908,
0x2b19191908190808, 0x2b19191919080808, 0x2b1919192b192b08, 0x2b1919192b2b0819,
0x2b19192b08080808, 0x2b19192b1908192b, 0x2b19192b192b1908, 0x2b192b0808080819,
0x2b192b0808081908, 0x2b192b0808190808, 0x2b192b08082b192b, 0x2b192b0819080808,
0x2b192b082b2b2b19, 0x2b192b1908080808, 0x2b192b1919082b19, 0x2b192b191919082b,
0x2b192b2b2b190808, 0x2b2b080808080808, 0x2b2b080808081919, 0x2b2b080808082b2b,
0x2b2b080808191908, 0x2b2b0808082b082b, 0x2b2b0808082b2b2b, 0x2b2b080819080819,
0x2b2b080819081908, 0x2b2b080819190808, 0x2b2b08082b2b082b, 0x2b2b08082b2b2b2b,
0x2b2b081919080808, 0x2b2b0819192b1919, 0x2b2b082b0808082b, 0x2b2b082b08082b2b,
0x2b2b082b082b082b, 0x2b2b082b082b2b08, 0x2b2b082b082b2b2b, 0x2b2b082b2b08082b,
0x2b2b082b2b082b08, 0x2b2b082b2b082b2b, 0x2b2b082b2b2b2b08, 0x2b2b190808080819,
0x2b2b190808081908, 0x2b2b190808190808, 0x2b2b190819080808, 0x2b2b19082b082b19,
0x2b2b19082b2b1908, 0x2b2b191908080808, 0x2b2b191908192b19, 0x2b2b192b19190819,
0x2b2b2b0808082b2b, 0x2b2b2b08082b2b08, 0x2b2b2b082b2b082b, 0x2b2b2b1919191908,
0x2b2b2b192b08192b, 0x2b2b2b2b08082b08, 0x2b2b2b2b08082b2b, 0x2b2b2b2b082b0808,
0x2b2b2b2b082b082b, 0x2b2b2b2b082b2b08, 0x2b2b2b2b2b082b08, 0x2b2b2b2b2b2b2b2b,
GGML_TABLE_END()
GGML_TABLE_BEGIN(uint32_t, iq3xxs_grid, 256)
0x04040404, 0x04040414, 0x04040424, 0x04040c0c, 0x04040c1c, 0x04040c3e, 0x04041404, 0x04041414,
0x04041c0c, 0x04042414, 0x04043e1c, 0x04043e2c, 0x040c040c, 0x040c041c, 0x040c0c04, 0x040c0c14,
0x040c140c, 0x040c142c, 0x040c1c04, 0x040c1c14, 0x040c240c, 0x040c2c24, 0x040c3e04, 0x04140404,
0x04140414, 0x04140424, 0x04140c0c, 0x04141404, 0x04141414, 0x04141c0c, 0x04141c1c, 0x04141c3e,
0x04142c0c, 0x04142c3e, 0x04143e2c, 0x041c040c, 0x041c043e, 0x041c0c04, 0x041c0c14, 0x041c142c,
0x041c3e04, 0x04240c1c, 0x04241c3e, 0x04242424, 0x04242c3e, 0x04243e1c, 0x04243e2c, 0x042c040c,
0x042c043e, 0x042c1c14, 0x042c2c14, 0x04341c2c, 0x04343424, 0x043e0c04, 0x043e0c24, 0x043e0c34,
0x043e241c, 0x043e340c, 0x0c04040c, 0x0c04041c, 0x0c040c04, 0x0c040c14, 0x0c04140c, 0x0c04141c,
0x0c041c04, 0x0c041c14, 0x0c041c24, 0x0c04243e, 0x0c042c04, 0x0c0c0404, 0x0c0c0414, 0x0c0c0c0c,
0x0c0c1404, 0x0c0c1414, 0x0c14040c, 0x0c14041c, 0x0c140c04, 0x0c140c14, 0x0c14140c, 0x0c141c04,
0x0c143e14, 0x0c1c0404, 0x0c1c0414, 0x0c1c1404, 0x0c1c1c0c, 0x0c1c2434, 0x0c1c3434, 0x0c24040c,
0x0c24042c, 0x0c242c04, 0x0c2c1404, 0x0c2c1424, 0x0c2c2434, 0x0c2c3e0c, 0x0c34042c, 0x0c3e1414,
0x0c3e2404, 0x14040404, 0x14040414, 0x14040c0c, 0x14040c1c, 0x14041404, 0x14041414, 0x14041434,
0x14041c0c, 0x14042414, 0x140c040c, 0x140c041c, 0x140c042c, 0x140c0c04, 0x140c0c14, 0x140c140c,
0x140c1c04, 0x140c341c, 0x140c343e, 0x140c3e04, 0x14140404, 0x14140414, 0x14140c0c, 0x14140c3e,
0x14141404, 0x14141414, 0x14141c3e, 0x14142404, 0x14142c2c, 0x141c040c, 0x141c0c04, 0x141c0c24,
0x141c3e04, 0x141c3e24, 0x14241c2c, 0x14242c1c, 0x142c041c, 0x142c143e, 0x142c240c, 0x142c3e24,
0x143e040c, 0x143e041c, 0x143e0c34, 0x143e242c, 0x1c04040c, 0x1c040c04, 0x1c040c14, 0x1c04140c,
0x1c04141c, 0x1c042c04, 0x1c04342c, 0x1c043e14, 0x1c0c0404, 0x1c0c0414, 0x1c0c1404, 0x1c0c1c0c,
0x1c0c2424, 0x1c0c2434, 0x1c14040c, 0x1c14041c, 0x1c140c04, 0x1c14142c, 0x1c142c14, 0x1c143e14,
0x1c1c0c0c, 0x1c1c1c1c, 0x1c241c04, 0x1c24243e, 0x1c243e14, 0x1c2c0404, 0x1c2c0434, 0x1c2c1414,
0x1c2c2c2c, 0x1c340c24, 0x1c341c34, 0x1c34341c, 0x1c3e1c1c, 0x1c3e3404, 0x24040424, 0x24040c3e,
0x24041c2c, 0x24041c3e, 0x24042c1c, 0x24042c3e, 0x240c3e24, 0x24141404, 0x24141c3e, 0x24142404,
0x24143404, 0x24143434, 0x241c043e, 0x241c242c, 0x24240424, 0x24242c0c, 0x24243424, 0x242c142c,
0x242c241c, 0x242c3e04, 0x243e042c, 0x243e0c04, 0x243e0c14, 0x243e1c04, 0x2c040c14, 0x2c04240c,
0x2c043e04, 0x2c0c0404, 0x2c0c0434, 0x2c0c1434, 0x2c0c2c2c, 0x2c140c24, 0x2c141c14, 0x2c143e14,
0x2c1c0414, 0x2c1c2c1c, 0x2c240c04, 0x2c24141c, 0x2c24143e, 0x2c243e14, 0x2c2c0414, 0x2c2c1c0c,
0x2c342c04, 0x2c3e1424, 0x2c3e2414, 0x34041424, 0x34042424, 0x34042434, 0x34043424, 0x340c140c,
0x340c340c, 0x34140c3e, 0x34143424, 0x341c1c04, 0x341c1c34, 0x34242424, 0x342c042c, 0x342c2c14,
0x34341c1c, 0x343e041c, 0x343e140c, 0x3e04041c, 0x3e04042c, 0x3e04043e, 0x3e040c04, 0x3e041c14,
0x3e042c14, 0x3e0c1434, 0x3e0c2404, 0x3e140c14, 0x3e14242c, 0x3e142c14, 0x3e1c0404, 0x3e1c0c2c,
0x3e1c1c1c, 0x3e1c3404, 0x3e24140c, 0x3e24240c, 0x3e2c0404, 0x3e2c0414, 0x3e2c1424, 0x3e341c04,
GGML_TABLE_END()
GGML_TABLE_BEGIN(uint32_t, iq3s_grid, 512)
0x01010101, 0x01010103, 0x01010105, 0x0101010b, 0x0101010f, 0x01010301, 0x01010303, 0x01010305,
0x01010309, 0x0101030d, 0x01010501, 0x01010503, 0x0101050b, 0x01010707, 0x01010901, 0x01010905,
0x0101090b, 0x0101090f, 0x01010b03, 0x01010b07, 0x01010d01, 0x01010d05, 0x01010f03, 0x01010f09,
0x01010f0f, 0x01030101, 0x01030103, 0x01030105, 0x01030109, 0x01030301, 0x01030303, 0x0103030b,
0x01030501, 0x01030507, 0x0103050f, 0x01030703, 0x0103070b, 0x01030909, 0x01030d03, 0x01030d0b,
0x01030f05, 0x01050101, 0x01050103, 0x0105010b, 0x0105010f, 0x01050301, 0x01050307, 0x0105030d,
0x01050503, 0x0105050b, 0x01050701, 0x01050709, 0x01050905, 0x0105090b, 0x0105090f, 0x01050b03,
0x01050b07, 0x01050f01, 0x01050f07, 0x01070107, 0x01070303, 0x0107030b, 0x01070501, 0x01070505,
0x01070703, 0x01070707, 0x0107070d, 0x01070909, 0x01070b01, 0x01070b05, 0x01070d0f, 0x01070f03,
0x01070f0b, 0x01090101, 0x01090307, 0x0109030f, 0x01090503, 0x01090509, 0x01090705, 0x01090901,
0x01090907, 0x01090b03, 0x01090f01, 0x010b0105, 0x010b0109, 0x010b0501, 0x010b0505, 0x010b050d,
0x010b0707, 0x010b0903, 0x010b090b, 0x010b090f, 0x010b0d0d, 0x010b0f07, 0x010d010d, 0x010d0303,
0x010d0307, 0x010d0703, 0x010d0b05, 0x010d0f03, 0x010f0101, 0x010f0105, 0x010f0109, 0x010f0501,
0x010f0505, 0x010f050d, 0x010f0707, 0x010f0b01, 0x010f0b09, 0x03010101, 0x03010103, 0x03010105,
0x03010109, 0x03010301, 0x03010303, 0x03010307, 0x0301030b, 0x0301030f, 0x03010501, 0x03010505,
0x03010703, 0x03010709, 0x0301070d, 0x03010b09, 0x03010b0d, 0x03010d03, 0x03010f05, 0x03030101,
0x03030103, 0x03030107, 0x0303010d, 0x03030301, 0x03030309, 0x03030503, 0x03030701, 0x03030707,
0x03030903, 0x03030b01, 0x03030b05, 0x03030f01, 0x03030f0d, 0x03050101, 0x03050305, 0x0305030b,
0x0305030f, 0x03050501, 0x03050509, 0x03050705, 0x03050901, 0x03050907, 0x03050b0b, 0x03050d01,
0x03050f05, 0x03070103, 0x03070109, 0x0307010f, 0x03070301, 0x03070307, 0x03070503, 0x0307050f,
0x03070701, 0x03070709, 0x03070903, 0x03070d05, 0x03070f01, 0x03090107, 0x0309010b, 0x03090305,
0x03090309, 0x03090703, 0x03090707, 0x03090905, 0x0309090d, 0x03090b01, 0x03090b09, 0x030b0103,
0x030b0301, 0x030b0307, 0x030b0503, 0x030b0701, 0x030b0705, 0x030b0b03, 0x030d0501, 0x030d0509,
0x030d050f, 0x030d0909, 0x030d090d, 0x030f0103, 0x030f0107, 0x030f0301, 0x030f0305, 0x030f0503,
0x030f070b, 0x030f0903, 0x030f0d05, 0x030f0f01, 0x05010101, 0x05010103, 0x05010107, 0x0501010b,
0x0501010f, 0x05010301, 0x05010305, 0x05010309, 0x0501030d, 0x05010503, 0x05010507, 0x0501050f,
0x05010701, 0x05010705, 0x05010903, 0x05010907, 0x0501090b, 0x05010b01, 0x05010b05, 0x05010d0f,
0x05010f01, 0x05010f07, 0x05010f0b, 0x05030101, 0x05030105, 0x05030301, 0x05030307, 0x0503030f,
0x05030505, 0x0503050b, 0x05030703, 0x05030709, 0x05030905, 0x05030b03, 0x05050103, 0x05050109,
0x0505010f, 0x05050503, 0x05050507, 0x05050701, 0x0505070f, 0x05050903, 0x05050b07, 0x05050b0f,
0x05050f03, 0x05050f09, 0x05070101, 0x05070105, 0x0507010b, 0x05070303, 0x05070505, 0x05070509,
0x05070703, 0x05070707, 0x05070905, 0x05070b01, 0x05070d0d, 0x05090103, 0x0509010f, 0x05090501,
0x05090507, 0x05090705, 0x0509070b, 0x05090903, 0x05090f05, 0x05090f0b, 0x050b0109, 0x050b0303,
0x050b0505, 0x050b070f, 0x050b0901, 0x050b0b07, 0x050b0f01, 0x050d0101, 0x050d0105, 0x050d010f,
0x050d0503, 0x050d0b0b, 0x050d0d03, 0x050f010b, 0x050f0303, 0x050f050d, 0x050f0701, 0x050f0907,
0x050f0b01, 0x07010105, 0x07010303, 0x07010307, 0x0701030b, 0x0701030f, 0x07010505, 0x07010703,
0x07010707, 0x0701070b, 0x07010905, 0x07010909, 0x0701090f, 0x07010b03, 0x07010d07, 0x07010f03,
0x07030103, 0x07030107, 0x0703010b, 0x07030309, 0x07030503, 0x07030507, 0x07030901, 0x07030d01,
0x07030f05, 0x07030f0d, 0x07050101, 0x07050305, 0x07050501, 0x07050705, 0x07050709, 0x07050b01,
0x07070103, 0x07070301, 0x07070309, 0x07070503, 0x07070507, 0x0707050f, 0x07070701, 0x07070903,
0x07070907, 0x0707090f, 0x07070b0b, 0x07070f07, 0x07090107, 0x07090303, 0x0709030d, 0x07090505,
0x07090703, 0x07090b05, 0x07090d01, 0x07090d09, 0x070b0103, 0x070b0301, 0x070b0305, 0x070b050b,
0x070b0705, 0x070b0909, 0x070b0b0d, 0x070b0f07, 0x070d030d, 0x070d0903, 0x070f0103, 0x070f0107,
0x070f0501, 0x070f0505, 0x070f070b, 0x09010101, 0x09010109, 0x09010305, 0x09010501, 0x09010509,
0x0901050f, 0x09010705, 0x09010903, 0x09010b01, 0x09010f01, 0x09030105, 0x0903010f, 0x09030303,
0x09030307, 0x09030505, 0x09030701, 0x0903070b, 0x09030907, 0x09030b03, 0x09030b0b, 0x09050103,
0x09050107, 0x09050301, 0x0905030b, 0x09050503, 0x09050707, 0x09050901, 0x09050b0f, 0x09050d05,
0x09050f01, 0x09070109, 0x09070303, 0x09070307, 0x09070501, 0x09070505, 0x09070703, 0x0907070b,
0x09090101, 0x09090105, 0x09090509, 0x0909070f, 0x09090901, 0x09090f03, 0x090b010b, 0x090b010f,
0x090b0503, 0x090b0d05, 0x090d0307, 0x090d0709, 0x090d0d01, 0x090f0301, 0x090f030b, 0x090f0701,
0x090f0907, 0x090f0b03, 0x0b010105, 0x0b010301, 0x0b010309, 0x0b010505, 0x0b010901, 0x0b010909,
0x0b01090f, 0x0b010b05, 0x0b010d0d, 0x0b010f09, 0x0b030103, 0x0b030107, 0x0b03010b, 0x0b030305,
0x0b030503, 0x0b030705, 0x0b030f05, 0x0b050101, 0x0b050303, 0x0b050507, 0x0b050701, 0x0b05070d,
0x0b050b07, 0x0b070105, 0x0b07010f, 0x0b070301, 0x0b07050f, 0x0b070909, 0x0b070b03, 0x0b070d0b,
0x0b070f07, 0x0b090103, 0x0b090109, 0x0b090501, 0x0b090705, 0x0b09090d, 0x0b0b0305, 0x0b0b050d,
0x0b0b0b03, 0x0b0b0b07, 0x0b0d0905, 0x0b0f0105, 0x0b0f0109, 0x0b0f0505, 0x0d010303, 0x0d010307,
0x0d01030b, 0x0d010703, 0x0d010707, 0x0d010d01, 0x0d030101, 0x0d030501, 0x0d03050f, 0x0d030d09,
0x0d050305, 0x0d050709, 0x0d050905, 0x0d050b0b, 0x0d050d05, 0x0d050f01, 0x0d070101, 0x0d070309,
0x0d070503, 0x0d070901, 0x0d09050b, 0x0d090907, 0x0d090d05, 0x0d0b0101, 0x0d0b0107, 0x0d0b0709,
0x0d0b0d01, 0x0d0d010b, 0x0d0d0901, 0x0d0f0303, 0x0d0f0307, 0x0f010101, 0x0f010109, 0x0f01010f,
0x0f010501, 0x0f010505, 0x0f01070d, 0x0f010901, 0x0f010b09, 0x0f010d05, 0x0f030105, 0x0f030303,
0x0f030509, 0x0f030907, 0x0f03090b, 0x0f050103, 0x0f050109, 0x0f050301, 0x0f05030d, 0x0f050503,
0x0f050701, 0x0f050b03, 0x0f070105, 0x0f070705, 0x0f07070b, 0x0f070b07, 0x0f090103, 0x0f09010b,
0x0f090307, 0x0f090501, 0x0f090b01, 0x0f0b0505, 0x0f0b0905, 0x0f0d0105, 0x0f0d0703, 0x0f0f0101,
GGML_TABLE_END()
#define NGRID_IQ1S 2048
#define IQ1S_DELTA 0.125f
#define IQ1M_DELTA 0.125f
#if defined(GGML_COMMON_IMPL_C)
GGML_TABLE_BEGIN(uint64_t, iq1s_grid, NGRID_IQ1S)
0xffffffffffffffff, 0xffffffffffffff01, 0xffffffffffff0000, 0xffffffffffff01ff,
0xffffffffffff0101, 0xffffffffff00ff00, 0xffffffffff000000, 0xffffffffff01ffff,
0xffffffffff01ff01, 0xffffffffff0101ff, 0xffffffffff010101, 0xffffffff00ff0000,
0xffffffff0000ff00, 0xffffffff000000ff, 0xffffffff00000001, 0xffffffff00010000,
0xffffffff01ffffff, 0xffffffff01ffff01, 0xffffffff01ff01ff, 0xffffffff01ff0101,
0xffffffff01000000, 0xffffffff0101ffff, 0xffffffff0101ff01, 0xffffffff010101ff,
0xffffffff01010101, 0xffffff00ffff00ff, 0xffffff00ffff0000, 0xffffff00ff00ff00,
0xffffff00ff0000ff, 0xffffff00ff000001, 0xffffff00ff000100, 0xffffff00ff000101,
0xffffff00ff010000, 0xffffff0000ffff00, 0xffffff0000ff0001, 0xffffff0000ff0100,
0xffffff000000ff01, 0xffffff0000000000, 0xffffff0000000101, 0xffffff000001ff00,
0xffffff00000100ff, 0xffffff0000010001, 0xffffff00000101ff, 0xffffff0001ff0000,
0xffffff000100ff00, 0xffffff00010000ff, 0xffffff0001000001, 0xffffff0001010000,
0xffffff01ffffffff, 0xffffff01ffffff01, 0xffffff01ffff01ff, 0xffffff01ffff0101,
0xffffff01ff000000, 0xffffff01ff01ffff, 0xffffff01ff01ff01, 0xffffff01ff0101ff,
0xffffff01ff010101, 0xffffff0100ff0000, 0xffffff010000ff00, 0xffffff0100000100,
0xffffff01000100ff, 0xffffff0100010100, 0xffffff0101ffffff, 0xffffff0101ffff01,
0xffffff0101ff01ff, 0xffffff0101ff0101, 0xffffff010100ff00, 0xffffff0101000000,
0xffffff0101000100, 0xffffff010101ffff, 0xffffff010101ff01, 0xffffff01010101ff,
0xffffff0101010101, 0xffff00ffff00ff00, 0xffff00ffff0000ff, 0xffff00ffff000001,
0xffff00ffff010000, 0xffff00ff00ffff00, 0xffff00ff00ff0100, 0xffff00ff00000000,
0xffff00ff00000101, 0xffff00ff000100ff, 0xffff00ff00010000, 0xffff00ff0100ff00,
0xffff00ff01000100, 0xffff00ff01010000, 0xffff0000ffffff00, 0xffff0000ffff00ff,
0xffff0000ffff0000, 0xffff0000ffff0001, 0xffff0000ff000000, 0xffff0000ff0001ff,
0xffff0000ff000101, 0xffff0000ff010100, 0xffff000000ffffff, 0xffff000000ff0000,
0xffff000000ff0101, 0xffff00000000ffff, 0xffff00000000ff00, 0xffff0000000000ff,
0xffff000000000000, 0xffff000000000001, 0xffff000000000100, 0xffff00000001ffff,
0xffff00000001ff01, 0xffff000000010000, 0xffff0000000101ff, 0xffff000000010101,
0xffff000001ffff00, 0xffff00000100ff00, 0xffff000001000000, 0xffff0000010001ff,
0xffff000001000101, 0xffff00000101ff00, 0xffff0000010100ff, 0xffff000001010000,
0xffff000001010001, 0xffff000001010100, 0xffff0001ff0000ff, 0xffff0001ff000100,
0xffff000100ffff00, 0xffff000100ff00ff, 0xffff00010000ffff, 0xffff00010000ff01,
0xffff000100000000, 0xffff0001000001ff, 0xffff00010001ffff, 0xffff00010001ff00,
0xffff000100010001, 0xffff000100010100, 0xffff000101ff0000, 0xffff00010100ff00,
0xffff0001010000ff, 0xffff000101000100, 0xffff01ffffffffff, 0xffff01ffffffff01,
0xffff01ffffff01ff, 0xffff01ffffff0101, 0xffff01ffff000000, 0xffff01ffff01ffff,
0xffff01ffff01ff01, 0xffff01ffff0101ff, 0xffff01ffff010101, 0xffff01ff00ff0000,
0xffff01ff0000ff00, 0xffff01ff00000001, 0xffff01ff00010000, 0xffff01ff01ffffff,
0xffff01ff01ffff01, 0xffff01ff01ff01ff, 0xffff01ff01ff0101, 0xffff01ff01000000,
0xffff01ff0101ffff, 0xffff01ff0101ff01, 0xffff01ff010101ff, 0xffff01ff01010101,
0xffff0100ffff0000, 0xffff0100ff00ff00, 0xffff0100ff0000ff, 0xffff0100ff000100,
0xffff0100ff0100ff, 0xffff0100ff010000, 0xffff010000ffff00, 0xffff01000000ffff,
0xffff01000000ff00, 0xffff010000000000, 0xffff01000001ff00, 0xffff0100000100ff,
0xffff010000010100, 0xffff01000100ff00, 0xffff0100010000ff, 0xffff010001000001,
0xffff010001000100, 0xffff010001010000, 0xffff0101ffffffff, 0xffff0101ffffff01,
0xffff0101ffff01ff, 0xffff0101ffff0101, 0xffff0101ff000000, 0xffff0101ff01ffff,
0xffff0101ff01ff01, 0xffff0101ff0101ff, 0xffff0101ff010101, 0xffff010100ff0000,
0xffff01010000ff00, 0xffff010100000100, 0xffff01010001ff00, 0xffff010100010000,
0xffff010101ffffff, 0xffff010101ffff01, 0xffff010101ff0000, 0xffff010101ff01ff,
0xffff010101ff0101, 0xffff010101000000, 0xffff01010101ffff, 0xffff01010101ff01,
0xffff0101010101ff, 0xffff010101010101, 0xff00ffffff00ffff, 0xff00ffffff00ff00,
0xff00ffffff0000ff, 0xff00ffffff000100, 0xff00ffffff0100ff, 0xff00ffffff010000,
0xff00ffff00ffff00, 0xff00ffff00ff00ff, 0xff00ffff0000ffff, 0xff00ffff00000000,
0xff00ffff000001ff, 0xff00ffff0001ff00, 0xff00ffff000100ff, 0xff00ffff00010000,
0xff00ffff00010100, 0xff00ffff0100ff00, 0xff00ffff010000ff, 0xff00ffff01000001,
0xff00ffff0101ff00, 0xff00ffff01010000, 0xff00ff00ffffff00, 0xff00ff00ffff00ff,
0xff00ff00ffff0001, 0xff00ff00ffff0100, 0xff00ff00ff00ffff, 0xff00ff00ff00ff01,
0xff00ff00ff000000, 0xff00ff00ff0001ff, 0xff00ff00ff01ff00, 0xff00ff00ff0100ff,
0xff00ff00ff010100, 0xff00ff0000ff0000, 0xff00ff0000ff0101, 0xff00ff000000ffff,
0xff00ff000000ff00, 0xff00ff000000ff01, 0xff00ff00000000ff, 0xff00ff0000000000,
0xff00ff0000000001, 0xff00ff0000000100, 0xff00ff000001ffff, 0xff00ff0000010000,
0xff00ff0001ff00ff, 0xff00ff000100ff01, 0xff00ff0001000000, 0xff00ff000101ff00,
0xff00ff00010100ff, 0xff00ff01ff00ff00, 0xff00ff01ff0000ff, 0xff00ff01ff000001,
0xff00ff01ff010000, 0xff00ff0100ffffff, 0xff00ff0100ff0001, 0xff00ff0100ff0100,
0xff00ff010000ff01, 0xff00ff0100000000, 0xff00ff01000001ff, 0xff00ff0100000101,
0xff00ff01000100ff, 0xff00ff0100010001, 0xff00ff0101ff0000, 0xff00ff010100ff00,
0xff00ff01010000ff, 0xff00ff0101000001, 0xff00ff0101010000, 0xff0000ffffffff00,
0xff0000ffffff0001, 0xff0000ffffff0100, 0xff0000ffff0000ff, 0xff0000ffff000000,
0xff0000ffff0001ff, 0xff0000ffff000100, 0xff0000ffff01ff00, 0xff0000ffff010001,
0xff0000ff00ffff00, 0xff0000ff00ff0000, 0xff0000ff00ff0001, 0xff0000ff00ff01ff,
0xff0000ff00ff0101, 0xff0000ff0000ff00, 0xff0000ff000000ff, 0xff0000ff00000000,
0xff0000ff00000001, 0xff0000ff00000100, 0xff0000ff0001ff01, 0xff0000ff00010000,
0xff0000ff000101ff, 0xff0000ff01ff00ff, 0xff0000ff01ff0100, 0xff0000ff0100ffff,
0xff0000ff010000ff, 0xff0000ff01000000, 0xff0000ff010001ff, 0xff0000ff01000100,
0xff0000ff01000101, 0xff0000ff0101ff00, 0xff0000ff010100ff, 0xff0000ff01010000,
0xff0000ff01010100, 0xff000000ffffff01, 0xff000000ffff0000, 0xff000000ffff0101,
0xff000000ff00ff00, 0xff000000ff0000ff, 0xff000000ff000000, 0xff000000ff000001,
0xff000000ff000100, 0xff000000ff01ffff, 0xff000000ff01ff01, 0xff000000ff010000,
0xff000000ff0101ff, 0xff000000ff010101, 0xff00000000ffff00, 0xff00000000ff00ff,
0xff00000000ff0000, 0xff00000000ff0001, 0xff0000000000ff00, 0xff0000000000ff01,
0xff000000000000ff, 0xff00000000000000, 0xff00000000000001, 0xff00000000000100,
0xff00000000000101, 0xff0000000001ff00, 0xff000000000100ff, 0xff00000000010000,
0xff00000000010001, 0xff00000000010100, 0xff00000001ffffff, 0xff00000001ffff01,
0xff00000001ff00ff, 0xff00000001ff0000, 0xff00000001ff01ff, 0xff00000001ff0101,
0xff0000000100ffff, 0xff0000000100ff00, 0xff000000010000ff, 0xff00000001000000,
0xff00000001000001, 0xff00000001000100, 0xff00000001000101, 0xff0000000101ffff,
0xff0000000101ff01, 0xff00000001010000, 0xff000001ffffff00, 0xff000001ffff00ff,
0xff000001ffff0000, 0xff000001ffff0001, 0xff000001ff000000, 0xff000001ff000001,
0xff000001ff0001ff, 0xff000001ff000101, 0xff000001ff01ff00, 0xff000001ff010001,
0xff00000100ffffff, 0xff00000100ffff01, 0xff00000100ff00ff, 0xff00000100ff0000,
0xff00000100ff01ff, 0xff00000100ff0101, 0xff0000010000ff00, 0xff00000100000000,
0xff00000100000001, 0xff000001000001ff, 0xff00000100000100, 0xff0000010001ff00,
0xff000001000100ff, 0xff00000100010000, 0xff000001000101ff, 0xff00000100010100,
0xff00000100010101, 0xff00000101ff0001, 0xff00000101ff0101, 0xff0000010100ff01,
0xff00000101000000, 0xff000001010100ff, 0xff00000101010100, 0xff0001ffff00ff00,
0xff0001ffff000001, 0xff0001ffff010000, 0xff0001ff00ffff00, 0xff0001ff00ff00ff,
0xff0001ff00ff0001, 0xff0001ff00ff0100, 0xff0001ff0000ffff, 0xff0001ff00000000,
0xff0001ff000001ff, 0xff0001ff00000101, 0xff0001ff0001ffff, 0xff0001ff0001ff00,
0xff0001ff000100ff, 0xff0001ff00010001, 0xff0001ff00010100, 0xff0001ff01ff0000,
0xff0001ff0100ff00, 0xff0001ff010000ff, 0xff0001ff01010000, 0xff000100ff00ffff,
0xff000100ff00ff01, 0xff000100ff000000, 0xff000100ff000101, 0xff000100ff01ff00,
0xff000100ff010000, 0xff00010000ffff01, 0xff00010000ff00ff, 0xff00010000ff0000,
0xff00010000ff01ff, 0xff0001000000ff00, 0xff000100000000ff, 0xff00010000000000,
0xff00010000000001, 0xff00010000000100, 0xff00010000000101, 0xff0001000001ffff,
0xff00010000010000, 0xff00010000010101, 0xff00010001ff0100, 0xff0001000100ff00,
0xff0001000100ff01, 0xff00010001000000, 0xff000100010001ff, 0xff0001000101ff00,
0xff00010001010001, 0xff00010001010100, 0xff000101ffff0100, 0xff000101ff000001,
0xff000101ff0100ff, 0xff000101ff010001, 0xff00010100ff00ff, 0xff00010100ff0001,
0xff00010100ff0100, 0xff0001010000ffff, 0xff0001010000ff01, 0xff00010100000000,
0xff000101000001ff, 0xff0001010001ff00, 0xff00010100010001, 0xff00010100010100,
0xff00010101ff0000, 0xff0001010100ff00, 0xff00010101000001, 0xff00010101000101,
0xff01ffffffffffff, 0xff01ffffffffff01, 0xff01ffffffff01ff, 0xff01ffffffff0101,
0xff01ffffff000000, 0xff01ffffff01ffff, 0xff01ffffff01ff01, 0xff01ffffff010000,
0xff01ffffff0101ff, 0xff01ffffff010101, 0xff01ffff00ff0000, 0xff01ffff0000ff00,
0xff01ffff00000100, 0xff01ffff0001ff00, 0xff01ffff00010000, 0xff01ffff01ffffff,
0xff01ffff01ffff01, 0xff01ffff01ff01ff, 0xff01ffff01ff0101, 0xff01ffff01000000,
0xff01ffff0101ffff, 0xff01ffff0101ff01, 0xff01ffff01010000, 0xff01ffff010101ff,
0xff01ffff01010101, 0xff01ff00ffff0000, 0xff01ff00ff00ff00, 0xff01ff00ff0000ff,
0xff01ff00ff000100, 0xff01ff00ff010000, 0xff01ff0000ffff01, 0xff01ff0000ff00ff,
0xff01ff0000ff0100, 0xff01ff0000000000, 0xff01ff00000001ff, 0xff01ff0000000101,
0xff01ff000001ff00, 0xff01ff00000100ff, 0xff01ff0000010000, 0xff01ff0000010001,
0xff01ff0001ff0000, 0xff01ff000100ffff, 0xff01ff0001000001, 0xff01ff0001000100,
0xff01ff0001010000, 0xff01ff01ffffff00, 0xff01ff01ffff01ff, 0xff01ff01ffff0101,
0xff01ff01ff00ff00, 0xff01ff01ff000000, 0xff01ff01ff01ffff, 0xff01ff01ff01ff01,
0xff01ff01ff0101ff, 0xff01ff01ff010101, 0xff01ff0100ff0000, 0xff01ff010000ff00,
0xff01ff0100000001, 0xff01ff0100000100, 0xff01ff0100010000, 0xff01ff0101ffff00,
0xff01ff0101ff01ff, 0xff01ff0101ff0101, 0xff01ff010100ff00, 0xff01ff0101000000,
0xff01ff010101ffff, 0xff01ff010101ff01, 0xff01ff01010101ff, 0xff01ff0101010101,
0xff0100ffffff0000, 0xff0100ffff0000ff, 0xff0100ffff000001, 0xff0100ffff000100,
0xff0100ffff010000, 0xff0100ff00ff00ff, 0xff0100ff00ff0000, 0xff0100ff00ff0001,
0xff0100ff00ff0100, 0xff0100ff0000ff01, 0xff0100ff00000000, 0xff0100ff000001ff,
0xff0100ff00000101, 0xff0100ff00010001, 0xff0100ff01ff0000, 0xff0100ff0100ff00,
0xff0100ff010000ff, 0xff0100ff01000100, 0xff0100ff0101ff00, 0xff0100ff01010000,
0xff010000ffff0100, 0xff010000ff000000, 0xff010000ff01ff00, 0xff010000ff010100,
0xff01000000ffffff, 0xff01000000ff0000, 0xff01000000ff01ff, 0xff0100000000ff00,
0xff010000000000ff, 0xff01000000000000, 0xff01000000000100, 0xff0100000001ff01,
0xff01000000010000, 0xff010000000101ff, 0xff01000001ff0100, 0xff0100000100ffff,
0xff010000010000ff, 0xff01000001000000, 0xff010000010001ff, 0xff01000001000101,
0xff0100000101ff00, 0xff010000010100ff, 0xff01000001010001, 0xff01000001010100,
0xff010001ffff0000, 0xff010001ff00ffff, 0xff010001ff00ff01, 0xff010001ff000100,
0xff010001ff010000, 0xff01000100ffff00, 0xff01000100ff0100, 0xff01000100000000,
0xff0100010001ffff, 0xff0100010001ff00, 0xff01000100010100, 0xff01000101ff00ff,
0xff01000101ff0001, 0xff0100010100ffff, 0xff01000101000101, 0xff0101ffffffffff,
0xff0101ffffffff01, 0xff0101ffffff01ff, 0xff0101ffffff0101, 0xff0101ffff000000,
0xff0101ffff01ffff, 0xff0101ffff01ff01, 0xff0101ffff0101ff, 0xff0101ffff010101,
0xff0101ff00ff0000, 0xff0101ff0000ff00, 0xff0101ff000000ff, 0xff0101ff00010000,
0xff0101ff01ffffff, 0xff0101ff01ffff01, 0xff0101ff01ff01ff, 0xff0101ff01ff0101,
0xff0101ff0101ffff, 0xff0101ff0101ff01, 0xff0101ff010101ff, 0xff0101ff01010101,
0xff010100ffff0100, 0xff010100ff00ff00, 0xff010100ff0000ff, 0xff010100ff000100,
0xff010100ff010000, 0xff01010000ff0001, 0xff01010000ff0100, 0xff0101000000ff01,
0xff01010000000000, 0xff0101000001ff00, 0xff010100000100ff, 0xff01010000010001,
0xff01010000010100, 0xff01010001ff0000, 0xff0101000100ffff, 0xff01010001000001,
0xff01010001000100, 0xff010100010100ff, 0xff01010001010000, 0xff010101ffffffff,
0xff010101ffffff01, 0xff010101ffff01ff, 0xff010101ffff0101, 0xff010101ff01ffff,
0xff010101ff01ff01, 0xff010101ff0101ff, 0xff010101ff010101, 0xff01010100ff0000,
0xff0101010000ff00, 0xff01010100000001, 0xff01010100000100, 0xff01010100010000,
0xff01010101ffffff, 0xff01010101ffff01, 0xff01010101ff01ff, 0xff01010101ff0101,
0xff01010101000000, 0xff0101010101ffff, 0xff0101010101ff01, 0xff010101010101ff,
0xff01010101010101, 0x00ffffffffff0000, 0x00ffffffff00ff00, 0x00ffffffff000001,
0x00ffffffff010000, 0x00ffffff00ff0100, 0x00ffffff0000ff01, 0x00ffffff00000000,
0x00ffffff000001ff, 0x00ffffff00000101, 0x00ffffff0001ff00, 0x00ffffff000100ff,
0x00ffffff00010001, 0x00ffffff010000ff, 0x00ffffff01000100, 0x00ffffff0101ff00,
0x00ffffff01010001, 0x00ffff00ffffffff, 0x00ffff00ffffff00, 0x00ffff00ffff00ff,
0x00ffff00ffff0001, 0x00ffff00ffff0100, 0x00ffff00ff00ff01, 0x00ffff00ff000000,
0x00ffff00ff000001, 0x00ffff00ff0001ff, 0x00ffff00ff000101, 0x00ffff00ff01ff00,
0x00ffff00ff010001, 0x00ffff00ff010100, 0x00ffff0000ff0000, 0x00ffff0000ff01ff,
0x00ffff0000ff0101, 0x00ffff000000ff00, 0x00ffff00000000ff, 0x00ffff0000000000,
0x00ffff0000000001, 0x00ffff0000000100, 0x00ffff0000000101, 0x00ffff0000010000,
0x00ffff00000101ff, 0x00ffff0000010101, 0x00ffff0001ffff00, 0x00ffff0001ff00ff,
0x00ffff0001ff0001, 0x00ffff000100ffff, 0x00ffff000100ff01, 0x00ffff0001000000,
0x00ffff000101ffff, 0x00ffff000101ff00, 0x00ffff000101ff01, 0x00ffff01ffff0000,
0x00ffff01ff00ff00, 0x00ffff01ff0000ff, 0x00ffff01ff000001, 0x00ffff01ff010000,
0x00ffff0100ffff00, 0x00ffff010000ff01, 0x00ffff0100000000, 0x00ffff0100000101,
0x00ffff01000100ff, 0x00ffff0100010100, 0x00ffff0101ff0100, 0x00ffff01010000ff,
0x00ffff0101010000, 0x00ff00ffffffff00, 0x00ff00ffff000000, 0x00ff00ffff000100,
0x00ff00ffff010100, 0x00ff00ff00ff0000, 0x00ff00ff00ff01ff, 0x00ff00ff00ff0101,
0x00ff00ff0000ff00, 0x00ff00ff000000ff, 0x00ff00ff00000000, 0x00ff00ff00000001,
0x00ff00ff0001ff00, 0x00ff00ff0001ff01, 0x00ff00ff00010000, 0x00ff00ff000101ff,
0x00ff00ff00010101, 0x00ff00ff01ffff00, 0x00ff00ff01ff0001, 0x00ff00ff01ff0100,
0x00ff00ff0100ffff, 0x00ff00ff0100ff01, 0x00ff00ff01000000, 0x00ff00ff0101ffff,
0x00ff00ff0101ff00, 0x00ff00ff01010100, 0x00ff0000ffffff00, 0x00ff0000ffffff01,
0x00ff0000ffff0000, 0x00ff0000ffff0101, 0x00ff0000ff00ff00, 0x00ff0000ff0000ff,
0x00ff0000ff000000, 0x00ff0000ff000001, 0x00ff0000ff000100, 0x00ff0000ff01ffff,
0x00ff0000ff010000, 0x00ff0000ff010101, 0x00ff000000ffff00, 0x00ff000000ff00ff,
0x00ff000000ff0000, 0x00ff000000ff0001, 0x00ff000000ff0100, 0x00ff00000000ffff,
0x00ff00000000ff00, 0x00ff0000000000ff, 0x00ff000000000000, 0x00ff000000000001,
0x00ff0000000001ff, 0x00ff000000000100, 0x00ff00000001ff00, 0x00ff0000000100ff,
0x00ff000000010000, 0x00ff000000010001, 0x00ff000000010100, 0x00ff000001ffff01,
0x00ff000001ff00ff, 0x00ff000001ff0000, 0x00ff000001ff01ff, 0x00ff00000100ff00,
0x00ff0000010000ff, 0x00ff000001000000, 0x00ff000001000001, 0x00ff000001000100,
0x00ff000001000101, 0x00ff000001010000, 0x00ff0000010101ff, 0x00ff000001010101,
0x00ff0001ffffff00, 0x00ff0001ffff0000, 0x00ff0001ffff0100, 0x00ff0001ff0000ff,
0x00ff0001ff000000, 0x00ff0001ff0001ff, 0x00ff0001ff000101, 0x00ff0001ff01ff00,
0x00ff0001ff0100ff, 0x00ff0001ff010100, 0x00ff000100ffffff, 0x00ff000100ffff01,
0x00ff000100ff0000, 0x00ff000100ff01ff, 0x00ff00010000ffff, 0x00ff00010000ff00,
0x00ff00010000ff01, 0x00ff000100000000, 0x00ff000100000001, 0x00ff000100000100,
0x00ff00010001ff01, 0x00ff000100010000, 0x00ff0001000101ff, 0x00ff000101ffff00,
0x00ff000101ff0000, 0x00ff000101ff0101, 0x00ff0001010000ff, 0x00ff000101000000,
0x00ff00010101ff00, 0x00ff0001010100ff, 0x00ff000101010001, 0x00ff01ffffff0000,
0x00ff01ffff00ff00, 0x00ff01ffff000000, 0x00ff01ffff000101, 0x00ff01ffff010000,
0x00ff01ff00ffff01, 0x00ff01ff00ff0100, 0x00ff01ff0000ffff, 0x00ff01ff00000000,
0x00ff01ff000001ff, 0x00ff01ff0001ff00, 0x00ff01ff000100ff, 0x00ff01ff00010001,
0x00ff01ff00010100, 0x00ff01ff01ff0000, 0x00ff01ff0100ff00, 0x00ff01ff010000ff,
0x00ff01ff01000001, 0x00ff01ff01000100, 0x00ff01ff01010000, 0x00ff0100ffffff00,
0x00ff0100ffff0000, 0x00ff0100ffff0001, 0x00ff0100ffff0101, 0x00ff0100ff00ffff,
0x00ff0100ff0000ff, 0x00ff0100ff000000, 0x00ff0100ff0001ff, 0x00ff0100ff01ff00,
0x00ff0100ff0100ff, 0x00ff0100ff010001, 0x00ff010000ffffff, 0x00ff010000ff0000,
0x00ff010000ff0101, 0x00ff01000000ff00, 0x00ff01000000ff01, 0x00ff0100000000ff,
0x00ff010000000000, 0x00ff010000000001, 0x00ff010000000100, 0x00ff01000001ffff,
0x00ff01000001ff01, 0x00ff010000010000, 0x00ff010000010001, 0x00ff010000010101,
0x00ff010001ff0001, 0x00ff010001ff0100, 0x00ff01000100ff01, 0x00ff010001000000,
0x00ff010001000001, 0x00ff0100010001ff, 0x00ff01000101ff00, 0x00ff0100010100ff,
0x00ff010001010001, 0x00ff010001010100, 0x00ff0101ff000001, 0x00ff010100ff00ff,
0x00ff010100ff0001, 0x00ff010100ff0100, 0x00ff010100000000, 0x00ff0101000001ff,
0x00ff010100000101, 0x00ff0101000100ff, 0x00ff010100010100, 0x00ff0101010000ff,
0x00ff010101010000, 0x0000ffffffffff00, 0x0000ffffffff00ff, 0x0000ffffffff0000,
0x0000ffffffff0001, 0x0000ffffffff0100, 0x0000ffffff00ff01, 0x0000ffffff000000,
0x0000ffffff000101, 0x0000ffffff01ff00, 0x0000ffffff0100ff, 0x0000ffffff010100,
0x0000ffff00ffffff, 0x0000ffff00ff0000, 0x0000ffff00ff01ff, 0x0000ffff0000ff00,
0x0000ffff000000ff, 0x0000ffff00000000, 0x0000ffff00000001, 0x0000ffff00000100,
0x0000ffff00010000, 0x0000ffff000101ff, 0x0000ffff01ff0001, 0x0000ffff01ff0100,
0x0000ffff01000000, 0x0000ffff010001ff, 0x0000ffff0101ffff, 0x0000ffff0101ff00,
0x0000ffff01010001, 0x0000ffff01010100, 0x0000ff00ffff0000, 0x0000ff00ffff01ff,
0x0000ff00ffff0100, 0x0000ff00ffff0101, 0x0000ff00ff00ff00, 0x0000ff00ff0000ff,
0x0000ff00ff000000, 0x0000ff00ff000001, 0x0000ff00ff0001ff, 0x0000ff00ff000100,
0x0000ff00ff01ffff, 0x0000ff00ff010000, 0x0000ff00ff010001, 0x0000ff00ff0101ff,
0x0000ff00ff010101, 0x0000ff0000ffff00, 0x0000ff0000ff00ff, 0x0000ff0000ff0000,
0x0000ff0000ff0001, 0x0000ff0000ff0100, 0x0000ff000000ffff, 0x0000ff000000ff00,
0x0000ff000000ff01, 0x0000ff00000000ff, 0x0000ff0000000000, 0x0000ff0000000001,
0x0000ff00000001ff, 0x0000ff0000000100, 0x0000ff0000000101, 0x0000ff000001ff00,
0x0000ff00000100ff, 0x0000ff0000010000, 0x0000ff0000010001, 0x0000ff0000010100,
0x0000ff0001ffff01, 0x0000ff0001ff0000, 0x0000ff000100ff00, 0x0000ff00010000ff,
0x0000ff0001000000, 0x0000ff0001000001, 0x0000ff0001000100, 0x0000ff000101ffff,
0x0000ff0001010000, 0x0000ff0001010101, 0x0000ff01ffffff00, 0x0000ff01ffff0001,
0x0000ff01ff00ff01, 0x0000ff01ff000000, 0x0000ff01ff000101, 0x0000ff01ff01ff00,
0x0000ff01ff0100ff, 0x0000ff0100ffff01, 0x0000ff0100ff0000, 0x0000ff0100ff0101,
0x0000ff010000ff00, 0x0000ff01000000ff, 0x0000ff0100000000, 0x0000ff0100000001,
0x0000ff0100000100, 0x0000ff010001ff01, 0x0000ff0100010000, 0x0000ff0101ff0000,
0x0000ff010100ffff, 0x0000ff010100ff01, 0x0000ff0101000000, 0x0000ff0101000100,
0x0000ff0101000101, 0x0000ff01010100ff, 0x000000ffffff00ff, 0x000000ffffff0000,
0x000000ffff00ff00, 0x000000ffff0000ff, 0x000000ffff000000, 0x000000ffff000001,
0x000000ffff0001ff, 0x000000ffff000100, 0x000000ffff01ff00, 0x000000ffff010000,
0x000000ffff0101ff, 0x000000ffff010101, 0x000000ff00ffff00, 0x000000ff00ff00ff,
0x000000ff00ff0000, 0x000000ff00ff0001, 0x000000ff00ff0100, 0x000000ff00ff0101,
0x000000ff0000ffff, 0x000000ff0000ff00, 0x000000ff000000ff, 0x000000ff00000000,
0x000000ff00000001, 0x000000ff000001ff, 0x000000ff00000100, 0x000000ff00000101,
0x000000ff0001ff00, 0x000000ff0001ff01, 0x000000ff000100ff, 0x000000ff00010000,
0x000000ff00010001, 0x000000ff00010100, 0x000000ff01ffffff, 0x000000ff01ff01ff,
0x000000ff01ff0101, 0x000000ff0100ff00, 0x000000ff010000ff, 0x000000ff01000000,
0x000000ff01000001, 0x000000ff01000100, 0x000000ff0101ff00, 0x000000ff010100ff,
0x000000ff01010000, 0x000000ff01010101, 0x00000000ffffff00, 0x00000000ffffff01,
0x00000000ffff00ff, 0x00000000ffff0000, 0x00000000ffff0001, 0x00000000ffff0100,
0x00000000ff00ffff, 0x00000000ff00ff00, 0x00000000ff00ff01, 0x00000000ff0000ff,
0x00000000ff000000, 0x00000000ff000001, 0x00000000ff000100, 0x00000000ff000101,
0x00000000ff01ff00, 0x00000000ff0100ff, 0x00000000ff010000, 0x00000000ff010001,
0x00000000ff010100, 0x0000000000ffffff, 0x0000000000ffff00, 0x0000000000ffff01,
0x0000000000ff00ff, 0x0000000000ff0000, 0x0000000000ff0001, 0x0000000000ff01ff,
0x0000000000ff0100, 0x000000000000ffff, 0x000000000000ff00, 0x000000000000ff01,
0x00000000000000ff, 0x0000000000000000, 0x0000000000000001, 0x00000000000001ff,
0x0000000000000100, 0x0000000000000101, 0x000000000001ffff, 0x000000000001ff00,
0x00000000000100ff, 0x0000000000010000, 0x0000000000010001, 0x00000000000101ff,
0x0000000000010100, 0x0000000000010101, 0x0000000001ffff00, 0x0000000001ff00ff,
0x0000000001ff0000, 0x0000000001ff0100, 0x0000000001ff0101, 0x000000000100ffff,
0x000000000100ff00, 0x00000000010000ff, 0x0000000001000000, 0x0000000001000001,
0x00000000010001ff, 0x0000000001000100, 0x000000000101ff00, 0x00000000010100ff,
0x0000000001010000, 0x0000000001010001, 0x0000000001010100, 0x00000001ffffffff,
0x00000001ffffff00, 0x00000001ffffff01, 0x00000001ffff00ff, 0x00000001ffff0001,
0x00000001ffff01ff, 0x00000001ffff0100, 0x00000001ff00ff00, 0x00000001ff0000ff,
0x00000001ff000000, 0x00000001ff0001ff, 0x00000001ff000100, 0x00000001ff01ffff,
0x00000001ff01ff00, 0x00000001ff01ff01, 0x00000001ff0100ff, 0x00000001ff010000,
0x00000001ff010001, 0x00000001ff0101ff, 0x00000001ff010100, 0x0000000100ffff00,
0x0000000100ff0000, 0x0000000100ff0001, 0x0000000100ff01ff, 0x0000000100ff0100,
0x0000000100ff0101, 0x000000010000ffff, 0x000000010000ff00, 0x000000010000ff01,
0x00000001000000ff, 0x0000000100000000, 0x0000000100000001, 0x00000001000001ff,
0x0000000100000100, 0x0000000100000101, 0x000000010001ff00, 0x00000001000100ff,
0x0000000100010000, 0x0000000100010100, 0x0000000101ffff01, 0x0000000101ff0000,
0x0000000101ff0001, 0x0000000101ff01ff, 0x0000000101ff0100, 0x0000000101ff0101,
0x000000010100ff00, 0x0000000101000000, 0x0000000101000101, 0x000000010101ff01,
0x0000000101010000, 0x0000000101010001, 0x00000001010101ff, 0x0000000101010100,
0x000001ffffff00ff, 0x000001ffffff0000, 0x000001ffffff0001, 0x000001ffffff0100,
0x000001ffff00ffff, 0x000001ffff000000, 0x000001ffff0001ff, 0x000001ffff01ff00,
0x000001ffff010101, 0x000001ff00ff0000, 0x000001ff00ff01ff, 0x000001ff00ff0101,
0x000001ff0000ff00, 0x000001ff000000ff, 0x000001ff00000000, 0x000001ff00000001,
0x000001ff000001ff, 0x000001ff00000100, 0x000001ff0001ffff, 0x000001ff0001ff01,
0x000001ff000100ff, 0x000001ff00010000, 0x000001ff01ffff01, 0x000001ff01ff0100,
0x000001ff0100ffff, 0x000001ff0100ff01, 0x000001ff01000000, 0x000001ff010001ff,
0x000001ff0101ff00, 0x000001ff01010100, 0x00000100ffffff00, 0x00000100ffffff01,
0x00000100ffff0000, 0x00000100ffff0101, 0x00000100ff00ff00, 0x00000100ff0000ff,
0x00000100ff000000, 0x00000100ff000001, 0x00000100ff000100, 0x00000100ff010000,
0x0000010000ffff00, 0x0000010000ff00ff, 0x0000010000ff0000, 0x0000010000ff0001,
0x0000010000ff0100, 0x000001000000ffff, 0x000001000000ff00, 0x000001000000ff01,
0x00000100000000ff, 0x0000010000000000, 0x0000010000000001, 0x00000100000001ff,
0x0000010000000100, 0x0000010000000101, 0x000001000001ff00, 0x00000100000100ff,
0x0000010000010000, 0x0000010000010001, 0x0000010000010100, 0x0000010001ffff00,
0x0000010001ff0000, 0x0000010001ff0100, 0x000001000100ff00, 0x00000100010000ff,
0x0000010001000000, 0x0000010001000001, 0x00000100010001ff, 0x0000010001000100,
0x0000010001010000, 0x00000101ffff00ff, 0x00000101ffff01ff, 0x00000101ff000000,
0x00000101ff000101, 0x00000101ff01ffff, 0x00000101ff010000, 0x00000101ff010001,
0x00000101ff010100, 0x0000010100ff0000, 0x0000010100ff01ff, 0x0000010100ff0100,
0x000001010000ff00, 0x0000010100000000, 0x0000010100000001, 0x00000101000001ff,
0x0000010100000100, 0x000001010001ff01, 0x0000010100010000, 0x00000101000101ff,
0x0000010100010101, 0x0000010101ffff00, 0x0000010101ff0101, 0x000001010100ff01,
0x0000010101000000, 0x0000010101000001, 0x00000101010001ff, 0x0000010101000101,
0x000001010101ff00, 0x0001ffffffff0000, 0x0001ffffff0000ff, 0x0001ffffff000001,
0x0001ffffff000100, 0x0001ffffff010000, 0x0001ffff00ff00ff, 0x0001ffff0000ffff,
0x0001ffff00000000, 0x0001ffff00000001, 0x0001ffff000001ff, 0x0001ffff00000101,
0x0001ffff0001ff00, 0x0001ffff000100ff, 0x0001ffff00010001, 0x0001ffff00010100,
0x0001ffff01ffff00, 0x0001ffff01000001, 0x0001ffff01010000, 0x0001ff00ffffff00,
0x0001ff00ffff00ff, 0x0001ff00ffff0001, 0x0001ff00ffff0100, 0x0001ff00ff00ff01,
0x0001ff00ff000000, 0x0001ff00ff01ff00, 0x0001ff00ff01ff01, 0x0001ff00ff010001,
0x0001ff00ff010100, 0x0001ff0000ff0000, 0x0001ff0000ff0100, 0x0001ff000000ff00,
0x0001ff0000000000, 0x0001ff0000000001, 0x0001ff0000000100, 0x0001ff0000010000,
0x0001ff0000010001, 0x0001ff0000010101, 0x0001ff0001ff00ff, 0x0001ff0001ff0101,
0x0001ff000100ff01, 0x0001ff0001000000, 0x0001ff000101ff00, 0x0001ff0001010001,
0x0001ff0001010100, 0x0001ff01ff00ff00, 0x0001ff01ff000001, 0x0001ff01ff000100,
0x0001ff0100ffffff, 0x0001ff0100ffff00, 0x0001ff0100ff0001, 0x0001ff0100000000,
0x0001ff0100000001, 0x0001ff01000001ff, 0x0001ff010001ffff, 0x0001ff0101ff0000,
0x0001ff010100ff00, 0x0001ff0101000001, 0x0001ff0101010000, 0x000100ffff00ff00,
0x000100ffff00ff01, 0x000100ffff000000, 0x000100ffff000001, 0x000100ffff000101,
0x000100ffff01ff00, 0x000100ffff010001, 0x000100ffff010100, 0x000100ff00ffffff,
0x000100ff00ffff01, 0x000100ff00ff0000, 0x000100ff00ff01ff, 0x000100ff00ff0101,
0x000100ff0000ff00, 0x000100ff000000ff, 0x000100ff00000000, 0x000100ff00000001,
0x000100ff00000100, 0x000100ff00000101, 0x000100ff0001ffff, 0x000100ff0001ff01,
0x000100ff00010000, 0x000100ff01ff00ff, 0x000100ff01ff0000, 0x000100ff01ff0100,
0x000100ff0100ffff, 0x000100ff0100ff01, 0x000100ff010000ff, 0x000100ff01000000,
0x000100ff01000001, 0x000100ff010001ff, 0x000100ff01000101, 0x000100ff0101ff00,
0x000100ff010100ff, 0x000100ff01010100, 0x00010000ffff0000, 0x00010000ffff01ff,
0x00010000ffff0101, 0x00010000ff00ff00, 0x00010000ff000000, 0x00010000ff000001,
0x00010000ff000100, 0x0001000000ff00ff, 0x0001000000ff0000, 0x0001000000ff0001,
0x0001000000ff0100, 0x000100000000ffff, 0x000100000000ff00, 0x00010000000000ff,
0x0001000000000000, 0x0001000000000001, 0x0001000000000100, 0x000100000001ff00,
0x00010000000100ff, 0x0001000000010000, 0x0001000000010001, 0x0001000000010100,
0x0001000001ff0001, 0x0001000001ff0100, 0x0001000001ff0101, 0x000100000100ff00,
0x0001000001000000, 0x0001000001000001, 0x0001000001000100, 0x0001000001000101,
0x000100000101ff01, 0x0001000001010000, 0x0001000001010001, 0x00010000010101ff,
0x00010001ffffff01, 0x00010001ffff0100, 0x00010001ff000000, 0x00010001ff01ffff,
0x00010001ff010001, 0x00010001ff0101ff, 0x00010001ff010100, 0x0001000100ffffff,
0x0001000100ff0000, 0x0001000100ff01ff, 0x0001000100ff0101, 0x000100010000ff00,
0x00010001000000ff, 0x0001000100000000, 0x0001000100000001, 0x00010001000001ff,
0x0001000100000101, 0x000100010001ffff, 0x0001000100010000, 0x00010001000101ff,
0x0001000101ffffff, 0x0001000101ffff01, 0x0001000101ff0000, 0x0001000101ff0101,
0x00010001010000ff, 0x0001000101000001, 0x00010001010001ff, 0x0001000101000100,
0x000100010101ffff, 0x00010001010100ff, 0x0001000101010001, 0x0001000101010101,
0x000101ffff000001, 0x000101ffff000100, 0x000101ffff010000, 0x000101ff00ffff00,
0x000101ff0000ff01, 0x000101ff00000000, 0x000101ff00000101, 0x000101ff0001ff00,
0x000101ff00010100, 0x000101ff01ff0000, 0x000101ff0100ff00, 0x000101ff010001ff,
0x000101ff01010001, 0x00010100ffffff00, 0x00010100ffff00ff, 0x00010100ff00ffff,
0x00010100ff000000, 0x00010100ff01ff00, 0x00010100ff0100ff, 0x00010100ff010001,
0x00010100ff010100, 0x0001010000ffffff, 0x0001010000ffff00, 0x0001010000ff0000,
0x0001010000ff0001, 0x0001010000ff01ff, 0x000101000000ff00, 0x00010100000000ff,
0x0001010000000000, 0x0001010000000001, 0x0001010000000100, 0x000101000001ffff,
0x0001010000010000, 0x0001010000010101, 0x0001010001ffff01, 0x0001010001ff00ff,
0x0001010001ff0101, 0x0001010001000000, 0x000101000101ff00, 0x00010100010100ff,
0x0001010001010000, 0x0001010001010100, 0x00010101ff00ff00, 0x00010101ff000001,
0x00010101ff0001ff, 0x0001010100ffff00, 0x0001010100ff00ff, 0x0001010100ff0100,
0x000101010000ffff, 0x0001010100000000, 0x00010101000001ff, 0x0001010100000101,
0x00010101000100ff, 0x0001010100010000, 0x0001010100010100, 0x0001010101ff0001,
0x00010101010000ff, 0x00010101010001ff, 0x0001010101000101, 0x0001010101010001,
0x01ffffffffffffff, 0x01ffffffffffff01, 0x01ffffffffff01ff, 0x01ffffffffff0101,
0x01ffffffff01ffff, 0x01ffffffff01ff01, 0x01ffffffff0101ff, 0x01ffffffff010101,
0x01ffffff00ff0000, 0x01ffffff0000ffff, 0x01ffffff0000ff00, 0x01ffffff000000ff,
0x01ffffff00000001, 0x01ffffff00000100, 0x01ffffff00010000, 0x01ffffff01ffffff,
0x01ffffff01ffff01, 0x01ffffff01ff01ff, 0x01ffffff01ff0101, 0x01ffffff01000000,
0x01ffffff0101ffff, 0x01ffffff0101ff01, 0x01ffffff010101ff, 0x01ffffff01010101,
0x01ffff00ffff0000, 0x01ffff00ff00ff00, 0x01ffff00ff0000ff, 0x01ffff00ff000001,
0x01ffff00ff000100, 0x01ffff00ff010000, 0x01ffff0000ffff00, 0x01ffff0000ff00ff,
0x01ffff0000ff0100, 0x01ffff000000ffff, 0x01ffff000000ff01, 0x01ffff0000000000,
0x01ffff0000000001, 0x01ffff00000001ff, 0x01ffff0000000100, 0x01ffff00000100ff,
0x01ffff0000010001, 0x01ffff0000010100, 0x01ffff0001ff0000, 0x01ffff0001ff0100,
0x01ffff00010000ff, 0x01ffff0001000001, 0x01ffff0001000100, 0x01ffff0001010000,
0x01ffff01ffffffff, 0x01ffff01ffffff01, 0x01ffff01ffff01ff, 0x01ffff01ffff0101,
0x01ffff01ff000000, 0x01ffff01ff01ffff, 0x01ffff01ff01ff01, 0x01ffff01ff0101ff,
0x01ffff01ff010101, 0x01ffff010000ff00, 0x01ffff01000000ff, 0x01ffff0100000100,
0x01ffff0100010000, 0x01ffff0101ffffff, 0x01ffff0101ffff01, 0x01ffff0101ff01ff,
0x01ffff0101ff0101, 0x01ffff0101000000, 0x01ffff010101ffff, 0x01ffff010101ff01,
0x01ffff01010101ff, 0x01ffff0101010101, 0x01ff00ffff0000ff, 0x01ff00ffff000100,
0x01ff00ff00ffff00, 0x01ff00ff00ff00ff, 0x01ff00ff0000ff00, 0x01ff00ff00000000,
0x01ff00ff00000101, 0x01ff00ff0001ff00, 0x01ff00ff000100ff, 0x01ff00ff00010100,
0x01ff00ff010000ff, 0x01ff00ff01000100, 0x01ff0000ffffff00, 0x01ff0000ffff0100,
0x01ff0000ff00ff01, 0x01ff0000ff000000, 0x01ff0000ff000101, 0x01ff0000ff010001,
0x01ff0000ff010100, 0x01ff000000ffffff, 0x01ff000000ffff00, 0x01ff000000ff0000,
0x01ff000000ff01ff, 0x01ff00000000ff00, 0x01ff0000000000ff, 0x01ff000000000000,
0x01ff000000000001, 0x01ff000000000100, 0x01ff000000000101, 0x01ff000000010000,
0x01ff000000010001, 0x01ff0000000101ff, 0x01ff000000010101, 0x01ff000001ffff00,
0x01ff000001ff00ff, 0x01ff000001ff0001, 0x01ff000001ff0100, 0x01ff00000100ffff,
0x01ff00000100ff01, 0x01ff000001000000, 0x01ff0000010001ff, 0x01ff000001010001,
0x01ff0001ff00ff00, 0x01ff0001ff000001, 0x01ff0001ff000100, 0x01ff0001ff010000,
0x01ff000100ffff00, 0x01ff000100ff00ff, 0x01ff000100ff0100, 0x01ff000100ff0101,
0x01ff00010000ffff, 0x01ff000100000000, 0x01ff000100000100, 0x01ff000100000101,
0x01ff00010001ff00, 0x01ff000100010001, 0x01ff000100010101, 0x01ff000101ff0000,
0x01ff00010100ff00, 0x01ff000101000101, 0x01ff0001010100ff, 0x01ff01ffffffffff,
0x01ff01ffffffff01, 0x01ff01ffffff01ff, 0x01ff01ffffff0101, 0x01ff01ffff000000,
0x01ff01ffff01ffff, 0x01ff01ffff01ff01, 0x01ff01ffff0101ff, 0x01ff01ffff010101,
0x01ff01ff00ffff00, 0x01ff01ff00ff0000, 0x01ff01ff0000ff00, 0x01ff01ff000000ff,
0x01ff01ff00000100, 0x01ff01ff00010000, 0x01ff01ff00010100, 0x01ff01ff01ffffff,
0x01ff01ff01ffff01, 0x01ff01ff01ff01ff, 0x01ff01ff01ff0101, 0x01ff01ff01000000,
0x01ff01ff0101ffff, 0x01ff01ff0101ff01, 0x01ff01ff010101ff, 0x01ff01ff01010101,
0x01ff0100ffff0000, 0x01ff0100ffff0001, 0x01ff0100ff00ff00, 0x01ff0100ff0000ff,
0x01ff0100ff000001, 0x01ff0100ff010000, 0x01ff010000ffff00, 0x01ff010000ff00ff,
0x01ff010000ff0001, 0x01ff010000ff0100, 0x01ff01000000ffff, 0x01ff01000000ff01,
0x01ff010000000000, 0x01ff010000000101, 0x01ff01000001ff00, 0x01ff0100000100ff,
0x01ff010001ff0000, 0x01ff010001000001, 0x01ff010001000100, 0x01ff010001010000,
0x01ff0101ffffffff, 0x01ff0101ffffff01, 0x01ff0101ffff01ff, 0x01ff0101ffff0101,
0x01ff0101ff000000, 0x01ff0101ff01ffff, 0x01ff0101ff01ff01, 0x01ff0101ff0101ff,
0x01ff0101ff010101, 0x01ff010100ff0000, 0x01ff01010000ff00, 0x01ff0101000000ff,
0x01ff010100000001, 0x01ff010101ffffff, 0x01ff010101ffff01, 0x01ff010101ff01ff,
0x01ff010101ff0101, 0x01ff010101000000, 0x01ff01010101ffff, 0x01ff01010101ff01,
0x01ff0101010101ff, 0x01ff010101010101, 0x0100ffffffff0000, 0x0100ffffff00ff00,
0x0100ffffff000001, 0x0100ffffff0001ff, 0x0100ffffff000100, 0x0100ffffff010000,
0x0100ffff00ffff00, 0x0100ffff00ff0001, 0x0100ffff00ff0100, 0x0100ffff00000000,
0x0100ffff000001ff, 0x0100ffff00000101, 0x0100ffff00010100, 0x0100ffff00010101,
0x0100ffff01ff0000, 0x0100ffff0100ff00, 0x0100ffff010000ff, 0x0100ffff01000001,
0x0100ffff01000100, 0x0100ffff01010000, 0x0100ff00ffffff00, 0x0100ff00ffff00ff,
0x0100ff00ffff0001, 0x0100ff00ffff0100, 0x0100ff00ff00ffff, 0x0100ff00ff000000,
0x0100ff00ff0001ff, 0x0100ff00ff000101, 0x0100ff00ff01ff00, 0x0100ff00ff0100ff,
0x0100ff00ff010001, 0x0100ff00ff010100, 0x0100ff0000ffffff, 0x0100ff0000ff0000,
0x0100ff000000ffff, 0x0100ff000000ff00, 0x0100ff00000000ff, 0x0100ff0000000000,
0x0100ff0000000001, 0x0100ff0000000100, 0x0100ff000001ff01, 0x0100ff0000010000,
0x0100ff0001ff00ff, 0x0100ff0001ff0001, 0x0100ff000100ff01, 0x0100ff0001000000,
0x0100ff00010001ff, 0x0100ff000101ff00, 0x0100ff00010100ff, 0x0100ff0001010001,
0x0100ff0001010100, 0x0100ff01ffff0000, 0x0100ff01ff00ff00, 0x0100ff01ff0000ff,
0x0100ff01ff000100, 0x0100ff01ff010000, 0x0100ff0100ff00ff, 0x0100ff0100ff0001,
0x0100ff0100ff0100, 0x0100ff010000ffff, 0x0100ff010000ff01, 0x0100ff0100000000,
0x0100ff01000001ff, 0x0100ff0100010001, 0x0100ff0100010100, 0x0100ff0101ff0000,
0x0100ff01010000ff, 0x0100ff0101000001, 0x0100ff0101010100, 0x010000ffffffff00,
0x010000ffffff00ff, 0x010000ffffff0001, 0x010000ffff00ffff, 0x010000ffff000000,
0x010000ffff0001ff, 0x010000ffff010001, 0x010000ff00ffffff, 0x010000ff00ff0101,
0x010000ff0000ff00, 0x010000ff000000ff, 0x010000ff00000000, 0x010000ff00000001,
0x010000ff000001ff, 0x010000ff00000100, 0x010000ff0001ffff, 0x010000ff0001ff00,
0x010000ff0001ff01, 0x010000ff00010000, 0x010000ff01ff00ff, 0x010000ff01ff0001,
0x010000ff0100ff01, 0x010000ff010000ff, 0x010000ff01000000, 0x010000ff010001ff,
0x010000ff0101ff00, 0x010000ff01010100, 0x01000000ffffffff, 0x01000000ffff0000,
0x01000000ffff01ff, 0x01000000ffff0101, 0x01000000ff00ffff, 0x01000000ff00ff00,
0x01000000ff0000ff, 0x01000000ff000000, 0x01000000ff000001, 0x01000000ff000100,
0x01000000ff01ff00, 0x01000000ff010000, 0x01000000ff010100, 0x01000000ff010101,
0x0100000000ffff00, 0x0100000000ff00ff, 0x0100000000ff0000, 0x0100000000ff0001,
0x0100000000ff0100, 0x010000000000ffff, 0x010000000000ff00, 0x010000000000ff01,
0x01000000000000ff, 0x0100000000000000, 0x0100000000000001, 0x01000000000001ff,
0x0100000000000100, 0x0100000000000101, 0x010000000001ff00, 0x01000000000100ff,
0x0100000000010000, 0x0100000000010001, 0x0100000000010100, 0x0100000001ffff00,
0x0100000001ff0000, 0x0100000001ff01ff, 0x010000000100ff00, 0x010000000100ff01,
0x01000000010000ff, 0x0100000001000000, 0x0100000001000001, 0x0100000001000100,
0x0100000001000101, 0x010000000101ffff, 0x010000000101ff01, 0x0100000001010000,
0x01000000010101ff, 0x0100000001010101, 0x01000001ffffff00, 0x01000001ffff00ff,
0x01000001ff00ffff, 0x01000001ff000000, 0x01000001ff000100, 0x01000001ff01ffff,
0x01000001ff010001, 0x01000001ff010100, 0x0100000100ff0000, 0x0100000100ff01ff,
0x0100000100ff0100, 0x010000010000ff00, 0x010000010000ff01, 0x0100000100000000,
0x0100000100000001, 0x0100000100000100, 0x0100000100010000, 0x01000001000101ff,
0x0100000101ffff01, 0x0100000101ff00ff, 0x0100000101ff0100, 0x0100000101ff0101,
0x010000010100ff01, 0x01000001010000ff, 0x0100000101000000, 0x01000001010100ff,
0x0100000101010001, 0x0100000101010100, 0x010001ffffff0000, 0x010001ffff000001,
0x010001ffff000100, 0x010001ffff010000, 0x010001ff00ffff00, 0x010001ff00ff0001,
0x010001ff0000ffff, 0x010001ff0000ff01, 0x010001ff00000000, 0x010001ff00000001,
0x010001ff00000101, 0x010001ff000100ff, 0x010001ff00010000, 0x010001ff01ff0000,
0x010001ff0100ff00, 0x010001ff01000001, 0x010001ff01000100, 0x010001ff01010000,
0x01000100ffff00ff, 0x01000100ffff0001, 0x01000100ffff0100, 0x01000100ff00ffff,
0x01000100ff00ff01, 0x01000100ff000000, 0x01000100ff0001ff, 0x01000100ff000101,
0x01000100ff01ffff, 0x01000100ff01ff00, 0x01000100ff0100ff, 0x01000100ff010001,
0x0100010000ffffff, 0x0100010000ffff01, 0x0100010000ff0000, 0x0100010000ff01ff,
0x0100010000ff0101, 0x010001000000ff00, 0x01000100000000ff, 0x0100010000000000,
0x0100010000000001, 0x0100010000000100, 0x010001000001ff01, 0x0100010000010000,
0x0100010000010001, 0x0100010000010101, 0x0100010001ffff00, 0x0100010001ff00ff,
0x010001000100ffff, 0x010001000100ff01, 0x0100010001000000, 0x0100010001000101,
0x010001000101ff00, 0x0100010001010001, 0x01000101ffff0000, 0x01000101ff000000,
0x01000101ff010000, 0x0100010100ff00ff, 0x0100010100ff0001, 0x0100010100ff0100,
0x010001010000ffff, 0x0100010100000000, 0x01000101000001ff, 0x010001010001ff00,
0x0100010101ff0000, 0x010001010100ff00, 0x01000101010000ff, 0x0100010101000000,
0x0100010101000001, 0x0101ffffffffffff, 0x0101ffffffffff01, 0x0101ffffffff01ff,
0x0101ffffffff0101, 0x0101ffffff000000, 0x0101ffffff01ffff, 0x0101ffffff01ff01,
0x0101ffffff0101ff, 0x0101ffffff010101, 0x0101ffff00ff0000, 0x0101ffff0000ff00,
0x0101ffff000000ff, 0x0101ffff00000001, 0x0101ffff00000100, 0x0101ffff01ffffff,
0x0101ffff01ffff01, 0x0101ffff01ff01ff, 0x0101ffff01ff0101, 0x0101ffff01000000,
0x0101ffff0101ffff, 0x0101ffff0101ff01, 0x0101ffff010101ff, 0x0101ffff01010101,
0x0101ff00ffff0000, 0x0101ff00ffff0100, 0x0101ff00ff00ff00, 0x0101ff00ff0000ff,
0x0101ff00ff000001, 0x0101ff00ff000100, 0x0101ff00ff000101, 0x0101ff0000ff0001,
0x0101ff0000ff0100, 0x0101ff000000ff00, 0x0101ff0000000000, 0x0101ff00000001ff,
0x0101ff0000000101, 0x0101ff000001ff00, 0x0101ff00000100ff, 0x0101ff0001ff0000,
0x0101ff000100ffff, 0x0101ff000100ff01, 0x0101ff0001000001, 0x0101ff0001000100,
0x0101ff01ffffff01, 0x0101ff01ffff01ff, 0x0101ff01ffff0101, 0x0101ff01ff00ffff,
0x0101ff01ff000100, 0x0101ff01ff01ff01, 0x0101ff01ff0101ff, 0x0101ff01ff010101,
0x0101ff0100ff0000, 0x0101ff010000ff00, 0x0101ff0100000001, 0x0101ff0100000100,
0x0101ff0100010000, 0x0101ff0101ffffff, 0x0101ff0101ffff01, 0x0101ff0101ff01ff,
0x0101ff0101ff0101, 0x0101ff0101000000, 0x0101ff010101ffff, 0x0101ff010101ff01,
0x0101ff01010101ff, 0x0101ff0101010101, 0x010100ffff000100, 0x010100ffff010000,
0x010100ff00ffff00, 0x010100ff00ff00ff, 0x010100ff0000ffff, 0x010100ff000000ff,
0x010100ff00000000, 0x010100ff000001ff, 0x010100ff00000101, 0x010100ff0001ff00,
0x010100ff00010000, 0x010100ff00010001, 0x010100ff000101ff, 0x010100ff00010100,
0x010100ff01ff0000, 0x01010000ffff0001, 0x01010000ffff0100, 0x01010000ff00ffff,
0x01010000ff00ff01, 0x01010000ff000000, 0x01010000ff0001ff, 0x01010000ff010001,
0x01010000ff010100, 0x0101000000ffff01, 0x0101000000ff0000, 0x010100000000ff00,
0x01010000000000ff, 0x0101000000000000, 0x0101000000000001, 0x0101000000000100,
0x0101000000010000, 0x0101000000010101, 0x0101000001ffff00, 0x0101000001ff00ff,
0x0101000001ff0000, 0x0101000001ff0001, 0x0101000001ff0100, 0x010100000100ff01,
0x0101000001000000, 0x01010000010001ff, 0x01010001ffff0000, 0x01010001ff00ff00,
0x01010001ff000001, 0x01010001ff000101, 0x01010001ff01ff00, 0x01010001ff010000,
0x0101000100ff00ff, 0x0101000100ff0001, 0x0101000100ff0101, 0x010100010000ff01,
0x0101000100000000, 0x0101000100000001, 0x01010001000001ff, 0x010100010001ffff,
0x010100010001ff01, 0x0101000101ff0001, 0x010100010100ffff, 0x0101000101000000,
0x0101000101000001, 0x0101000101000100, 0x010100010101ff00, 0x01010001010100ff,
0x0101000101010001, 0x010101ffffffffff, 0x010101ffffffff01, 0x010101ffffff01ff,
0x010101ffffff0101, 0x010101ffff01ffff, 0x010101ffff01ff01, 0x010101ffff0101ff,
0x010101ffff010101, 0x010101ff0000ff00, 0x010101ff000000ff, 0x010101ff00000001,
0x010101ff00000100, 0x010101ff01ffffff, 0x010101ff01ffff01, 0x010101ff01ff01ff,
0x010101ff01ff0101, 0x010101ff01000000, 0x010101ff0101ffff, 0x010101ff0101ff01,
0x010101ff010101ff, 0x010101ff01010101, 0x01010100ffff0000, 0x01010100ff0000ff,
0x01010100ff000100, 0x01010100ff01ff00, 0x01010100ff010000, 0x0101010000ffff00,
0x010101000000ffff, 0x0101010000000000, 0x0101010000000101, 0x010101000001ff00,
0x0101010000010001, 0x0101010000010100, 0x010101000100ffff, 0x0101010001000001,
0x01010101ffffffff, 0x01010101ffffff01, 0x01010101ffff01ff, 0x01010101ffff0101,
0x01010101ff01ffff, 0x01010101ff01ff01, 0x01010101ff0101ff, 0x01010101ff010101,
0x010101010000ff00, 0x01010101000000ff, 0x0101010100000001, 0x0101010101ffffff,
0x0101010101ffff01, 0x0101010101ff01ff, 0x0101010101ff0101, 0x0101010101000000,
0x010101010101ffff, 0x010101010101ff01, 0x01010101010101ff, 0x0101010101010101,
GGML_TABLE_END()
#else
GGML_TABLE_BEGIN(uint32_t, iq1s_grid_gpu, NGRID_IQ1S)
0x00000000, 0x00000002, 0x00000101, 0x00000200, 0x00000202, 0x00010001, 0x00010101, 0x00020000,
0x00020002, 0x00020200, 0x00020202, 0x01000101, 0x01010001, 0x01010100, 0x01010102, 0x01020101,
0x02000000, 0x02000002, 0x02000200, 0x02000202, 0x02010101, 0x02020000, 0x02020002, 0x02020200,
0x02020202, 0x00000110, 0x00000111, 0x00010011, 0x00010110, 0x00010112, 0x00010211, 0x00010212,
0x00020111, 0x01000011, 0x01000112, 0x01000211, 0x01010012, 0x01010111, 0x01010212, 0x01020011,
0x01020110, 0x01020112, 0x01020210, 0x02000111, 0x02010011, 0x02010110, 0x02010112, 0x02020111,
0x00000020, 0x00000022, 0x00000220, 0x00000222, 0x00010121, 0x00020020, 0x00020022, 0x00020220,
0x00020222, 0x01000121, 0x01010021, 0x01010221, 0x01020120, 0x01020221, 0x02000020, 0x02000022,
0x02000220, 0x02000222, 0x02010021, 0x02010121, 0x02010221, 0x02020020, 0x02020022, 0x02020220,
0x02020222, 0x00011001, 0x00011100, 0x00011102, 0x00021101, 0x01001001, 0x01001201, 0x01011101,
0x01011202, 0x01021100, 0x01021101, 0x02011001, 0x02011201, 0x02021101, 0x00001011, 0x00001110,
0x00001111, 0x00001112, 0x00011111, 0x00011210, 0x00011212, 0x00021211, 0x01001010, 0x01001111,
0x01001212, 0x01011010, 0x01011011, 0x01011110, 0x01011111, 0x01011112, 0x01011211, 0x01021010,
0x01021012, 0x01021111, 0x01021210, 0x01021212, 0x02001011, 0x02011011, 0x02011111, 0x02011210,
0x02011212, 0x02021011, 0x02021110, 0x02021111, 0x02021112, 0x02021211, 0x00011120, 0x00011221,
0x01001021, 0x01001120, 0x01011020, 0x01011022, 0x01011121, 0x01011220, 0x01021020, 0x01021021,
0x01021122, 0x01021221, 0x02001121, 0x02011021, 0x02011120, 0x02011221, 0x00002000, 0x00002002,
0x00002200, 0x00002202, 0x00012101, 0x00022000, 0x00022002, 0x00022200, 0x00022202, 0x01002101,
0x01012001, 0x01012102, 0x01022101, 0x02002000, 0x02002002, 0x02002200, 0x02002202, 0x02012101,
0x02022000, 0x02022002, 0x02022200, 0x02022202, 0x00002111, 0x00012011, 0x00012110, 0x00012211,
0x00022110, 0x00022111, 0x01002011, 0x01012010, 0x01012011, 0x01012111, 0x01022011, 0x01022110,
0x01022211, 0x02012011, 0x02012110, 0x02012112, 0x02012211, 0x02022111, 0x00002020, 0x00002022,
0x00002220, 0x00002222, 0x00012121, 0x00022020, 0x00022022, 0x00022220, 0x00022222, 0x01002121,
0x01012021, 0x01012221, 0x01022021, 0x01022121, 0x02002020, 0x02002022, 0x02002121, 0x02002220,
0x02002222, 0x02012121, 0x02022020, 0x02022022, 0x02022220, 0x02022222, 0x00110000, 0x00110001,
0x00110100, 0x00110201, 0x00120100, 0x00120101, 0x01100001, 0x01100100, 0x01110000, 0x01110101,
0x01110200, 0x01120001, 0x01120100, 0x01120101, 0x01120201, 0x02110001, 0x02110100, 0x02110102,
0x02120001, 0x02120101, 0x00100011, 0x00100110, 0x00100112, 0x00100211, 0x00110010, 0x00110012,
0x00110111, 0x00110210, 0x00120011, 0x00120110, 0x00120211, 0x01100111, 0x01100212, 0x01110010,
0x01110011, 0x01110012, 0x01110110, 0x01110111, 0x01110112, 0x01110211, 0x01120010, 0x01120111,
0x02100110, 0x02110012, 0x02110111, 0x02120011, 0x02120110, 0x00110021, 0x00110120, 0x00110122,
0x00120121, 0x01100020, 0x01100122, 0x01100221, 0x01110022, 0x01110121, 0x01110220, 0x01110222,
0x01120120, 0x01120122, 0x02100121, 0x02110021, 0x02110120, 0x02110122, 0x02120121, 0x00101001,
0x00101102, 0x00101201, 0x00111100, 0x00111101, 0x00111200, 0x00111201, 0x00121001, 0x00121102,
0x01101001, 0x01101101, 0x01101102, 0x01101200, 0x01101202, 0x01111001, 0x01111100, 0x01111101,
0x01111102, 0x01111201, 0x01121002, 0x01121101, 0x01121200, 0x02101100, 0x02101201, 0x02111000,
0x02111100, 0x02111101, 0x02111200, 0x02111201, 0x02111202, 0x02121001, 0x02121100, 0x02121101,
0x02121201, 0x00101012, 0x00101111, 0x00101212, 0x00111011, 0x00111110, 0x00111111, 0x00111112,
0x00111211, 0x00121010, 0x00121012, 0x00121111, 0x00121210, 0x00121212, 0x01101011, 0x01101110,
0x01101111, 0x01101112, 0x01111011, 0x01111012, 0x01111110, 0x01111111, 0x01111112, 0x01111211,
0x01111212, 0x01121011, 0x01121110, 0x01121111, 0x01121112, 0x01121211, 0x02101010, 0x02101012,
0x02101110, 0x02101111, 0x02101210, 0x02101212, 0x02111010, 0x02111011, 0x02111110, 0x02111111,
0x02111112, 0x02111211, 0x02111212, 0x02121010, 0x02121012, 0x02121111, 0x00101021, 0x00101120,
0x00101121, 0x00101122, 0x00111121, 0x00111122, 0x00111220, 0x00111222, 0x00121021, 0x00121122,
0x01101020, 0x01101022, 0x01101120, 0x01101121, 0x01101220, 0x01101222, 0x01111021, 0x01111121,
0x01111122, 0x01111220, 0x01111221, 0x01121021, 0x01121120, 0x01121121, 0x01121220, 0x01121221,
0x01121222, 0x02101122, 0x02101222, 0x02111022, 0x02111121, 0x02121120, 0x02121221, 0x00112001,
0x00112102, 0x00122101, 0x01102001, 0x01102100, 0x01102102, 0x01102201, 0x01112000, 0x01112101,
0x01112200, 0x01112202, 0x01122000, 0x01122001, 0x01122100, 0x01122102, 0x01122201, 0x02102101,
0x02112001, 0x02112100, 0x02122101, 0x00112010, 0x00112012, 0x00112111, 0x00112212, 0x00122011,
0x00122111, 0x01102012, 0x01102110, 0x01102111, 0x01102210, 0x01112011, 0x01112110, 0x01112111,
0x01112112, 0x01112211, 0x01112212, 0x01122010, 0x01122111, 0x01122212, 0x02102211, 0x02112011,
0x02112012, 0x02112111, 0x02112210, 0x02122011, 0x02122112, 0x02122211, 0x00102221, 0x00112122,
0x00122120, 0x00122122, 0x01102120, 0x01102122, 0x01102221, 0x01112020, 0x01112022, 0x01112121,
0x01112220, 0x01122021, 0x01122122, 0x01122221, 0x02102121, 0x02112021, 0x02112122, 0x02112222,
0x00200000, 0x00200002, 0x00200200, 0x00200202, 0x00210101, 0x00220000, 0x00220002, 0x00220101,
0x00220200, 0x00220202, 0x01200101, 0x01210001, 0x01210201, 0x01220001, 0x01220101, 0x02200000,
0x02200002, 0x02200200, 0x02200202, 0x02210101, 0x02220000, 0x02220002, 0x02220101, 0x02220200,
0x02220202, 0x00200111, 0x00210011, 0x00210110, 0x00210211, 0x00220111, 0x01200012, 0x01200110,
0x01200211, 0x01210111, 0x01210210, 0x01210212, 0x01220011, 0x01220110, 0x01220111, 0x01220112,
0x02200111, 0x02210010, 0x02210112, 0x02210211, 0x02220111, 0x00200021, 0x00200220, 0x00200222,
0x00210021, 0x00210121, 0x00220020, 0x00220022, 0x00220220, 0x00220222, 0x01200121, 0x01210021,
0x01210122, 0x01210221, 0x01220121, 0x02200021, 0x02200220, 0x02200222, 0x02210021, 0x02210121,
0x02220020, 0x02220022, 0x02220220, 0x02220222, 0x00201101, 0x00211100, 0x00211102, 0x00211201,
0x00221101, 0x01201100, 0x01201101, 0x01201102, 0x01201201, 0x01211002, 0x01211101, 0x01211200,
0x01211202, 0x01221102, 0x02201101, 0x02211001, 0x02211100, 0x02211201, 0x02221001, 0x02221101,
0x00201211, 0x00211111, 0x00221011, 0x00221211, 0x01201010, 0x01201111, 0x01201210, 0x01211011,
0x01211110, 0x01211111, 0x01211211, 0x01221012, 0x01221111, 0x01221210, 0x02201211, 0x02211010,
0x02211110, 0x02211111, 0x02211210, 0x02211212, 0x02221011, 0x02221110, 0x02221112, 0x02221211,
0x00201121, 0x00211020, 0x00211022, 0x00211221, 0x00221121, 0x01201021, 0x01201221, 0x01211121,
0x01221020, 0x01221021, 0x01221221, 0x02201120, 0x02201122, 0x02211020, 0x02211222, 0x00202000,
0x00202002, 0x00202200, 0x00202202, 0x00212101, 0x00222000, 0x00222002, 0x00222200, 0x00222202,
0x01202101, 0x01212001, 0x01212100, 0x01222101, 0x02202000, 0x02202002, 0x02202200, 0x02202202,
0x02222000, 0x02222002, 0x02222200, 0x02222202, 0x00202211, 0x00212011, 0x00212110, 0x00212211,
0x00222111, 0x01202112, 0x01202211, 0x01212012, 0x01212111, 0x01222011, 0x01222110, 0x01222112,
0x01222211, 0x02202111, 0x02212010, 0x02212112, 0x02212211, 0x02222110, 0x02222111, 0x00202020,
0x00202022, 0x00202220, 0x00202222, 0x00222020, 0x00222022, 0x00222220, 0x00222222, 0x01202121,
0x01212021, 0x01212122, 0x01212221, 0x01222121, 0x02202020, 0x02202022, 0x02202220, 0x02202222,
0x02212121, 0x02222020, 0x02222022, 0x02222220, 0x02222222, 0x10000101, 0x10010001, 0x10010102,
0x10020101, 0x11000201, 0x11010002, 0x11010101, 0x11010200, 0x11010202, 0x11020001, 0x11020100,
0x11020102, 0x12010100, 0x12010201, 0x12020001, 0x12020102, 0x10000010, 0x10000011, 0x10000110,
0x10000112, 0x10000211, 0x10010012, 0x10010111, 0x10010112, 0x10010210, 0x10010212, 0x10020011,
0x10020112, 0x10020211, 0x11000111, 0x11000210, 0x11000212, 0x11010011, 0x11010110, 0x11010111,
0x11010112, 0x11010211, 0x11010212, 0x11020111, 0x11020210, 0x11020212, 0x12000011, 0x12000110,
0x12000112, 0x12010010, 0x12010012, 0x12010111, 0x12020010, 0x12020011, 0x12020012, 0x10000121,
0x10010021, 0x10010120, 0x10010122, 0x10020121, 0x11000021, 0x11010022, 0x11010121, 0x11010222,
0x11020120, 0x11020221, 0x12000221, 0x12010120, 0x12020121, 0x10001001, 0x10011101, 0x10011201,
0x10021201, 0x11001101, 0x11001200, 0x11001202, 0x11011001, 0x11011100, 0x11011101, 0x11011102,
0x11021001, 0x11021002, 0x11021101, 0x11021200, 0x11021202, 0x12001001, 0x12001102, 0x12001201,
0x12011000, 0x12011002, 0x12011101, 0x12021000, 0x12021001, 0x12021201, 0x10001011, 0x10001012,
0x10001111, 0x10001212, 0x10011011, 0x10011110, 0x10011111, 0x10011112, 0x10011211, 0x10021010,
0x10021111, 0x10021212, 0x11001011, 0x11001110, 0x11001111, 0x11001112, 0x11001211, 0x11011010,
0x11011011, 0x11011110, 0x11011111, 0x11011112, 0x11011210, 0x11011211, 0x11021011, 0x11021110,
0x11021111, 0x11021112, 0x11021211, 0x12001012, 0x12001110, 0x12001111, 0x12001210, 0x12011011,
0x12011110, 0x12011111, 0x12011112, 0x12011211, 0x12011212, 0x12021111, 0x12021210, 0x12021212,
0x10001021, 0x10001121, 0x10001221, 0x10011120, 0x10011121, 0x10011220, 0x10011222, 0x10021021,
0x10021120, 0x10021221, 0x11001020, 0x11001022, 0x11001121, 0x11001220, 0x11011020, 0x11011021,
0x11011022, 0x11011121, 0x11011122, 0x11011221, 0x11021022, 0x11021121, 0x11021220, 0x12001021,
0x12001121, 0x12001222, 0x12011120, 0x12011121, 0x12021021, 0x12021120, 0x12021122, 0x10002101,
0x10012001, 0x10012101, 0x10012202, 0x10022101, 0x11002002, 0x11002201, 0x11012000, 0x11012101,
0x11012200, 0x11022001, 0x11022100, 0x11022102, 0x11022201, 0x12002101, 0x12012001, 0x12012100,
0x12012102, 0x12012201, 0x12022101, 0x10002011, 0x10002111, 0x10002112, 0x10002212, 0x10012010,
0x10012110, 0x10012111, 0x10012210, 0x10022011, 0x10022110, 0x10022112, 0x11002010, 0x11002111,
0x11002212, 0x11012011, 0x11012012, 0x11012110, 0x11012111, 0x11012112, 0x11012211, 0x11022010,
0x11022012, 0x11022111, 0x11022112, 0x11022212, 0x12002112, 0x12002211, 0x12012012, 0x12012111,
0x12012112, 0x12012210, 0x12022011, 0x12022110, 0x12022112, 0x12022211, 0x10012122, 0x11002120,
0x11002122, 0x11002221, 0x11012121, 0x11012220, 0x11012222, 0x11022120, 0x11022221, 0x12012120,
0x12022121, 0x10100001, 0x10100100, 0x10100101, 0x10100102, 0x10100201, 0x10110002, 0x10110101,
0x10110202, 0x10120001, 0x10120100, 0x10120201, 0x11100000, 0x11100101, 0x11100200, 0x11110001,
0x11110100, 0x11110101, 0x11110102, 0x11110201, 0x11120101, 0x11120200, 0x12100102, 0x12100201,
0x12110101, 0x12110200, 0x12120000, 0x12120001, 0x12120102, 0x12120201, 0x10100111, 0x10100210,
0x10100211, 0x10100212, 0x10110011, 0x10110110, 0x10110111, 0x10110112, 0x10110210, 0x10110211,
0x10120010, 0x10120111, 0x10120112, 0x10120210, 0x10120212, 0x11100011, 0x11100110, 0x11100111,
0x11100112, 0x11100211, 0x11110010, 0x11110011, 0x11110012, 0x11110110, 0x11110111, 0x11110112,
0x11110210, 0x11110211, 0x11110212, 0x11120011, 0x11120110, 0x11120111, 0x11120112, 0x11120211,
0x12100012, 0x12100111, 0x12110011, 0x12110110, 0x12110111, 0x12110112, 0x12110211, 0x12120010,
0x12120111, 0x12120212, 0x10100021, 0x10100122, 0x10110022, 0x10110121, 0x10110222, 0x10120021,
0x10120120, 0x11100022, 0x11100121, 0x11100222, 0x11110021, 0x11110120, 0x11110121, 0x11110122,
0x11110221, 0x11120022, 0x11120121, 0x12100121, 0x12110020, 0x12110022, 0x12110121, 0x12110221,
0x12110222, 0x12120120, 0x10101100, 0x10101101, 0x10111001, 0x10111100, 0x10111101, 0x10111102,
0x10111200, 0x10111201, 0x10121001, 0x10121101, 0x10121200, 0x10121202, 0x11101001, 0x11101100,
0x11101101, 0x11101102, 0x11101201, 0x11101202, 0x11111000, 0x11111001, 0x11111100, 0x11111101,
0x11111102, 0x11111200, 0x11111201, 0x11111202, 0x11121001, 0x11121002, 0x11121100, 0x11121101,
0x11121102, 0x11121201, 0x12101000, 0x12101200, 0x12101202, 0x12111001, 0x12111100, 0x12111101,
0x12111102, 0x12111201, 0x12121001, 0x12121100, 0x12121101, 0x12121202, 0x10101011, 0x10101012,
0x10101110, 0x10101111, 0x10101112, 0x10101211, 0x10111010, 0x10111011, 0x10111012, 0x10111110,
0x10111111, 0x10111112, 0x10111211, 0x10111212, 0x10121011, 0x10121110, 0x10121111, 0x10121112,
0x10121211, 0x11101010, 0x11101011, 0x11101012, 0x11101110, 0x11101111, 0x11101112, 0x11101210,
0x11101211, 0x11111010, 0x11111011, 0x11111012, 0x11111110, 0x11111111, 0x11111112, 0x11111210,
0x11111211, 0x11111212, 0x11121010, 0x11121011, 0x11121110, 0x11121111, 0x11121112, 0x11121210,
0x11121211, 0x11121212, 0x12101011, 0x12101110, 0x12101111, 0x12101211, 0x12101212, 0x12111010,
0x12111011, 0x12111110, 0x12111111, 0x12111112, 0x12111210, 0x12111211, 0x12121011, 0x12121110,
0x12121111, 0x12121112, 0x12121211, 0x10101020, 0x10101021, 0x10101022, 0x10101120, 0x10101122,
0x10101220, 0x10101221, 0x10111021, 0x10111120, 0x10111121, 0x10111220, 0x10111221, 0x10121020,
0x10121021, 0x10121022, 0x10121120, 0x10121121, 0x10121122, 0x10121220, 0x10121221, 0x11101021,
0x11101121, 0x11101122, 0x11101220, 0x11101221, 0x11101222, 0x11111020, 0x11111021, 0x11111022,
0x11111120, 0x11111121, 0x11111122, 0x11111220, 0x11111221, 0x11111222, 0x11121021, 0x11121120,
0x11121121, 0x11121221, 0x12101022, 0x12101121, 0x12101122, 0x12101220, 0x12101221, 0x12101222,
0x12111021, 0x12111121, 0x12111222, 0x12121022, 0x12121121, 0x12121122, 0x12121220, 0x12121221,
0x10102100, 0x10102101, 0x10102102, 0x10102201, 0x10112000, 0x10112101, 0x10112200, 0x10122001,
0x10122202, 0x11102101, 0x11102200, 0x11102202, 0x11112001, 0x11112100, 0x11112101, 0x11112102,
0x11112200, 0x11112201, 0x11122000, 0x11122002, 0x11122100, 0x11122101, 0x12102002, 0x12102201,
0x12112000, 0x12112002, 0x12112101, 0x12112200, 0x12122001, 0x12122201, 0x10102011, 0x10102012,
0x10102111, 0x10102212, 0x10112011, 0x10112110, 0x10112111, 0x10112112, 0x10112211, 0x10122111,
0x11102011, 0x11102110, 0x11102111, 0x11102112, 0x11102211, 0x11112010, 0x11112011, 0x11112012,
0x11112110, 0x11112111, 0x11112112, 0x11112210, 0x11112211, 0x11112212, 0x11122011, 0x11122110,
0x11122111, 0x11122112, 0x11122211, 0x12102011, 0x12102111, 0x12102211, 0x12112011, 0x12112110,
0x12112111, 0x12112112, 0x12112210, 0x12112211, 0x12122111, 0x10102120, 0x10102220, 0x10112121,
0x10112222, 0x10122020, 0x10122121, 0x10122122, 0x10122221, 0x11102121, 0x11102220, 0x11102221,
0x11112021, 0x11112121, 0x11112122, 0x11112220, 0x11112221, 0x11122022, 0x11122121, 0x11122220,
0x11122222, 0x12102021, 0x12102222, 0x12112022, 0x12112121, 0x12112122, 0x12112220, 0x12112222,
0x12122021, 0x10200101, 0x10210100, 0x10210102, 0x10210201, 0x10220101, 0x11200100, 0x11210000,
0x11210101, 0x11210102, 0x11210200, 0x11210202, 0x11220001, 0x11220100, 0x11220102, 0x11220201,
0x12200001, 0x12210102, 0x12220101, 0x10200011, 0x10200110, 0x10200112, 0x10200211, 0x10210012,
0x10210111, 0x10220011, 0x10220012, 0x10220112, 0x10220211, 0x11200111, 0x11200211, 0x11210011,
0x11210111, 0x11210112, 0x11210211, 0x11220111, 0x11220112, 0x11220212, 0x12200110, 0x12200212,
0x12210012, 0x12210111, 0x12220011, 0x12220112, 0x12220211, 0x10210021, 0x10210122, 0x10210221,
0x11200020, 0x11200021, 0x11200122, 0x11210121, 0x11210122, 0x11210220, 0x11220020, 0x12200121,
0x12210021, 0x12210122, 0x12220121, 0x10211001, 0x10211002, 0x10211101, 0x10211102, 0x10211202,
0x10221001, 0x10221102, 0x10221201, 0x11201000, 0x11201002, 0x11201101, 0x11201200, 0x11201202,
0x11211001, 0x11211100, 0x11211101, 0x11211102, 0x11211201, 0x11211202, 0x11221000, 0x11221002,
0x11221101, 0x12201100, 0x12201101, 0x12201201, 0x12211000, 0x12211002, 0x12211100, 0x12211101,
0x12211102, 0x12211200, 0x12211202, 0x12221001, 0x12221100, 0x12221201, 0x10201111, 0x10201210,
0x10201212, 0x10211011, 0x10211111, 0x10211112, 0x10211211, 0x11201110, 0x11201111, 0x11201112,
0x11201211, 0x11211010, 0x11211011, 0x11211110, 0x11211111, 0x11211112, 0x11211211, 0x11221011,
0x11221110, 0x11221111, 0x11221112, 0x11221211, 0x12201112, 0x12201211, 0x12201212, 0x12211011,
0x12211111, 0x12211112, 0x12211211, 0x12211212, 0x12221012, 0x12221111, 0x12221112, 0x12221210,
0x10201022, 0x10201221, 0x10211121, 0x10221020, 0x10221122, 0x10221220, 0x10221221, 0x11201020,
0x11201121, 0x11201220, 0x11201222, 0x11211021, 0x11211120, 0x11211121, 0x11211122, 0x11211220,
0x11211222, 0x11221020, 0x11221121, 0x11221220, 0x12201020, 0x12201022, 0x12201121, 0x12201222,
0x12211120, 0x12211122, 0x12211220, 0x12211221, 0x12221020, 0x12221120, 0x12221122, 0x12221222,
0x10212102, 0x10212201, 0x10222101, 0x11202001, 0x11212002, 0x11212101, 0x11212202, 0x11222001,
0x11222201, 0x12202101, 0x12212001, 0x12212200, 0x12222102, 0x10202011, 0x10202110, 0x10212010,
0x10212111, 0x10222011, 0x10222110, 0x10222112, 0x10222211, 0x11202010, 0x11202011, 0x11202111,
0x11202112, 0x11202210, 0x11212011, 0x11212110, 0x11212111, 0x11212112, 0x11212211, 0x11222010,
0x11222111, 0x11222212, 0x12202012, 0x12202110, 0x12202212, 0x12212111, 0x12222011, 0x12222110,
0x12222111, 0x12222211, 0x10212021, 0x10212122, 0x10212220, 0x11202021, 0x11202120, 0x11202221,
0x11212020, 0x11212121, 0x11212220, 0x11212222, 0x11222120, 0x11222121, 0x11222221, 0x12202122,
0x12212120, 0x12212220, 0x12212222, 0x12222122, 0x20000000, 0x20000002, 0x20000200, 0x20000202,
0x20020000, 0x20020002, 0x20020200, 0x20020202, 0x21000101, 0x21010000, 0x21010001, 0x21010100,
0x21010102, 0x21010201, 0x21020101, 0x22000000, 0x22000002, 0x22000200, 0x22000202, 0x22010101,
0x22020000, 0x22020002, 0x22020200, 0x22020202, 0x20000111, 0x20010011, 0x20010110, 0x20010112,
0x20010211, 0x20020111, 0x21000011, 0x21000110, 0x21000211, 0x21010010, 0x21010012, 0x21010111,
0x21010112, 0x21010210, 0x21010211, 0x21020110, 0x21020112, 0x21020211, 0x22000111, 0x22000211,
0x22010110, 0x22010112, 0x22010211, 0x22020111, 0x20000020, 0x20000022, 0x20000220, 0x20000222,
0x20010121, 0x20020020, 0x20020022, 0x20020220, 0x20020222, 0x21010021, 0x21010120, 0x21010221,
0x21020121, 0x22000020, 0x22000022, 0x22000220, 0x22000222, 0x22010121, 0x22020020, 0x22020022,
0x22020220, 0x22020222, 0x20011100, 0x20011201, 0x21001001, 0x21001100, 0x21011001, 0x21011101,
0x21011202, 0x21021001, 0x21021100, 0x21021201, 0x22011100, 0x22011201, 0x20001011, 0x20001211,
0x20011012, 0x20011111, 0x20011212, 0x20021112, 0x20021211, 0x21001010, 0x21001011, 0x21001111,
0x21001210, 0x21011011, 0x21011110, 0x21011111, 0x21011112, 0x21011211, 0x21011212, 0x21021111,
0x21021112, 0x21021210, 0x21021212, 0x22001011, 0x22001110, 0x22001112, 0x22001211, 0x22011010,
0x22011012, 0x22011111, 0x22011210, 0x22021112, 0x20011021, 0x20011122, 0x20011221, 0x20021121,
0x21001021, 0x21001120, 0x21001221, 0x21001222, 0x21011020, 0x21011121, 0x21011221, 0x21011222,
0x21021021, 0x21021122, 0x21021222, 0x22001121, 0x22011021, 0x22011222, 0x22021120, 0x20002000,
0x20002002, 0x20002200, 0x20002202, 0x20012101, 0x20022000, 0x20022002, 0x20022200, 0x20022202,
0x21002001, 0x21002101, 0x21012001, 0x21012100, 0x21012201, 0x21022101, 0x21022201, 0x22002000,
0x22002002, 0x22002200, 0x22002202, 0x22012101, 0x22022000, 0x22022002, 0x22022200, 0x22022202,
0x20002111, 0x20002112, 0x20012011, 0x20012110, 0x20012112, 0x20022111, 0x21002011, 0x21002110,
0x21002112, 0x21002211, 0x21012010, 0x21012012, 0x21012111, 0x21012212, 0x21022011, 0x21022110,
0x22002111, 0x22012112, 0x22012211, 0x22022111, 0x20002020, 0x20002022, 0x20002220, 0x20002222,
0x20012121, 0x20022020, 0x20022022, 0x20022220, 0x20022222, 0x21002121, 0x21012021, 0x21012120,
0x21012122, 0x22002020, 0x22002022, 0x22002220, 0x22002222, 0x22012121, 0x22022020, 0x22022022,
0x22022220, 0x22022222, 0x20100101, 0x20110001, 0x20110102, 0x20110200, 0x20110201, 0x20120101,
0x21100001, 0x21100102, 0x21100201, 0x21110101, 0x21110200, 0x21110202, 0x21120201, 0x21120202,
0x22100101, 0x22110001, 0x22110100, 0x22110102, 0x22110201, 0x22120101, 0x20100011, 0x20100110,
0x20100112, 0x20100211, 0x20110010, 0x20110111, 0x20110210, 0x20110212, 0x20120011, 0x20120110,
0x20120112, 0x20120211, 0x21100010, 0x21100111, 0x21110010, 0x21110011, 0x21110110, 0x21110111,
0x21110112, 0x21110211, 0x21120012, 0x21120111, 0x22100110, 0x22100112, 0x22110012, 0x22110111,
0x22110210, 0x22120011, 0x22120110, 0x22120112, 0x22120211, 0x20100121, 0x20110021, 0x20110120,
0x20110221, 0x20120121, 0x21100120, 0x21100122, 0x21100221, 0x21110020, 0x21110022, 0x21110121,
0x21110220, 0x21120122, 0x21120221, 0x22100121, 0x22110120, 0x22110122, 0x22120221, 0x20101001,
0x20101100, 0x20101102, 0x20111000, 0x20111101, 0x20111200, 0x20121102, 0x21101000, 0x21101202,
0x21111001, 0x21111100, 0x21111101, 0x21111102, 0x21111200, 0x21111201, 0x21121000, 0x21121001,
0x21121002, 0x21121101, 0x22101100, 0x22101102, 0x22111002, 0x22111100, 0x22111101, 0x22111200,
0x22121001, 0x22121201, 0x20101010, 0x20101111, 0x20101210, 0x20101212, 0x20111010, 0x20111011,
0x20111110, 0x20111111, 0x20111112, 0x20111211, 0x20121011, 0x20121111, 0x20121211, 0x20121212,
0x21101011, 0x21101110, 0x21101111, 0x21101112, 0x21101211, 0x21111010, 0x21111011, 0x21111012,
0x21111110, 0x21111111, 0x21111112, 0x21111210, 0x21111211, 0x21111212, 0x21121011, 0x21121110,
0x21121111, 0x21121112, 0x21121211, 0x22101011, 0x22101111, 0x22101210, 0x22111011, 0x22111012,
0x22111110, 0x22111111, 0x22111112, 0x22111211, 0x22111212, 0x22121010, 0x22121012, 0x22121111,
0x22121210, 0x22121212, 0x20101021, 0x20101120, 0x20111020, 0x20111121, 0x20111221, 0x20121020,
0x20121122, 0x20121221, 0x21101121, 0x21101220, 0x21101221, 0x21111021, 0x21111022, 0x21111121,
0x21111122, 0x21111221, 0x21121121, 0x21121220, 0x22101022, 0x22101120, 0x22101221, 0x22101222,
0x22111022, 0x22111120, 0x22111121, 0x22121120, 0x22121122, 0x22121221, 0x20102101, 0x20112102,
0x20112201, 0x20122101, 0x21102001, 0x21102102, 0x21112000, 0x21112002, 0x21112101, 0x21112102,
0x21112202, 0x21122100, 0x21122101, 0x22102101, 0x22112001, 0x22112102, 0x22112201, 0x22122101,
0x20102110, 0x20102112, 0x20102211, 0x20112010, 0x20112012, 0x20112111, 0x20112210, 0x20112212,
0x20122010, 0x20122011, 0x20122110, 0x20122112, 0x21102010, 0x21102012, 0x21102111, 0x21102210,
0x21102212, 0x21112011, 0x21112110, 0x21112111, 0x21112112, 0x21112211, 0x21122012, 0x21122111,
0x21122112, 0x21122212, 0x22102011, 0x22102110, 0x22112010, 0x22112012, 0x22112111, 0x22112212,
0x22122011, 0x22122112, 0x20102121, 0x20112121, 0x20122121, 0x21102120, 0x21102122, 0x21102221,
0x21112020, 0x21112121, 0x21112220, 0x21122021, 0x22102121, 0x22112021, 0x22112120, 0x22112121,
0x22112122, 0x20200000, 0x20200002, 0x20200200, 0x20200202, 0x20210101, 0x20220000, 0x20220002,
0x20220200, 0x20220202, 0x21200101, 0x21210001, 0x21210100, 0x21210102, 0x21210201, 0x22200000,
0x22200002, 0x22200200, 0x22200202, 0x22210101, 0x22220000, 0x22220002, 0x22220200, 0x22220202,
0x20200111, 0x20200211, 0x20210011, 0x20210110, 0x20210112, 0x20210211, 0x20210212, 0x21200112,
0x21200211, 0x21210011, 0x21210111, 0x21210210, 0x21210212, 0x21220011, 0x21220110, 0x22200111,
0x22210010, 0x22210012, 0x22210112, 0x22210211, 0x20200022, 0x20200220, 0x20200222, 0x20210020,
0x20210221, 0x20220022, 0x20220220, 0x20220222, 0x21200121, 0x21210021, 0x21210122, 0x21210221,
0x21220121, 0x22200020, 0x22200022, 0x22200220, 0x22200222, 0x22210121, 0x22220020, 0x22220022,
0x22220220, 0x22220222, 0x20211201, 0x20221101, 0x21201001, 0x21201100, 0x21211000, 0x21211100,
0x21211101, 0x21211200, 0x21211202, 0x21221001, 0x21221101, 0x21221102, 0x21221200, 0x21221201,
0x22201101, 0x20201112, 0x20201211, 0x20211010, 0x20211012, 0x20211111, 0x20211210, 0x20221112,
0x20221211, 0x21201012, 0x21201111, 0x21211011, 0x21211110, 0x21211111, 0x21211112, 0x21211211,
0x21221111, 0x21221212, 0x22201011, 0x22201110, 0x22201111, 0x22201112, 0x22201211, 0x22211012,
0x22211111, 0x22211210, 0x20201121, 0x20211021, 0x20211122, 0x20211222, 0x20221021, 0x20221121,
0x21201120, 0x21201122, 0x21201222, 0x21211022, 0x21211121, 0x21211122, 0x21211220, 0x21221020,
0x21221022, 0x22201122, 0x22211020, 0x22211121, 0x22211122, 0x22211221, 0x22221021, 0x22221120,
0x22221122, 0x20202000, 0x20202002, 0x20202200, 0x20202202, 0x20222000, 0x20222002, 0x20222200,
0x20222202, 0x21212001, 0x21212100, 0x21212102, 0x21212201, 0x22202000, 0x22202002, 0x22202200,
0x22202202, 0x22212101, 0x22222000, 0x22222002, 0x22222200, 0x22222202, 0x20202111, 0x20212110,
0x20212211, 0x20222011, 0x20222111, 0x21202011, 0x21212010, 0x21212111, 0x21212212, 0x21222011,
0x21222112, 0x21222211, 0x22212010, 0x22212112, 0x20202020, 0x20202022, 0x20202220, 0x20202222,
0x20222020, 0x20222022, 0x20222220, 0x20222222, 0x21212021, 0x21212120, 0x21212122, 0x22202020,
0x22202022, 0x22202220, 0x22202222, 0x22212121, 0x22222020, 0x22222022, 0x22222220, 0x22222222,
GGML_TABLE_END()
#endif
#endif // GGML_COMMON_IMPL
#endif // GGML_COMMON_IMPL
#include
using namespace metal;
#define MAX(x, y) ((x) > (y) ? (x) : (y))
#define MIN(x, y) ((x) < (y) ? (x) : (y))
#define SWAP(x, y) { auto tmp = (x); (x) = (y); (y) = tmp; }
#define N_SIMDWIDTH 32 // assuming SIMD group size is 32
enum ggml_sort_order {
GGML_SORT_ORDER_ASC,
GGML_SORT_ORDER_DESC,
};
// general-purpose kernel for addition, subtraction, multiplication and division of two tensors
// pros: works for non-contiguous tensors, supports broadcast across all dims
// cons: not very efficient
kernel void kernel_add(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant int64_t & offs,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig.z;
const int64_t i02 = tgpig.y;
const int64_t i01 = tgpig.x;
const int64_t i13 = i03 % ne13;
const int64_t i12 = i02 % ne12;
const int64_t i11 = i01 % ne11;
device const char * src0_ptr = src0 + i03*nb03 + i02*nb02 + i01*nb01 + offs;
device const char * src1_ptr = src1 + i13*nb13 + i12*nb12 + i11*nb11;
device char * dst_ptr = dst + i03*nb3 + i02*nb2 + i01*nb1 + offs;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int i10 = i0 % ne10;
*((device float *)(dst_ptr + i0*nb0)) = *((device float *)(src0_ptr + i0*nb00)) + *((device float *)(src1_ptr + i10*nb10));
}
}
kernel void kernel_sub(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant int64_t & offs,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig.z;
const int64_t i02 = tgpig.y;
const int64_t i01 = tgpig.x;
const int64_t i13 = i03 % ne13;
const int64_t i12 = i02 % ne12;
const int64_t i11 = i01 % ne11;
device const char * src0_ptr = src0 + i03*nb03 + i02*nb02 + i01*nb01 + offs;
device const char * src1_ptr = src1 + i13*nb13 + i12*nb12 + i11*nb11;
device char * dst_ptr = dst + i03*nb3 + i02*nb2 + i01*nb1 + offs;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int i10 = i0 % ne10;
*((device float *)(dst_ptr + i0*nb0)) = *((device float *)(src0_ptr + i0*nb00)) - *((device float *)(src1_ptr + i10*nb10));
}
}
kernel void kernel_mul(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig.z;
const int64_t i02 = tgpig.y;
const int64_t i01 = tgpig.x;
const int64_t i13 = i03 % ne13;
const int64_t i12 = i02 % ne12;
const int64_t i11 = i01 % ne11;
device const char * src0_ptr = src0 + i03*nb03 + i02*nb02 + i01*nb01;
device const char * src1_ptr = src1 + i13*nb13 + i12*nb12 + i11*nb11;
device char * dst_ptr = dst + i03*nb3 + i02*nb2 + i01*nb1;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int i10 = i0 % ne10;
*((device float *)(dst_ptr + i0*nb0)) = *((device float *)(src0_ptr + i0*nb00)) * *((device float *)(src1_ptr + i10*nb10));
}
}
kernel void kernel_div(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig.z;
const int64_t i02 = tgpig.y;
const int64_t i01 = tgpig.x;
const int64_t i13 = i03 % ne13;
const int64_t i12 = i02 % ne12;
const int64_t i11 = i01 % ne11;
device const char * src0_ptr = src0 + i03*nb03 + i02*nb02 + i01*nb01;
device const char * src1_ptr = src1 + i13*nb13 + i12*nb12 + i11*nb11;
device char * dst_ptr = dst + i03*nb3 + i02*nb2 + i01*nb1;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int i10 = i0 % ne10;
*((device float *)(dst_ptr + i0*nb0)) = *((device float *)(src0_ptr + i0*nb00)) / *((device float *)(src1_ptr + i10*nb10));
}
}
template
kernel void kernel_repeat(
device const char * src0,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i3 = tgpig.z;
const int64_t i2 = tgpig.y;
const int64_t i1 = tgpig.x;
const int64_t i03 = i3 % ne03;
const int64_t i02 = i2 % ne02;
const int64_t i01 = i1 % ne01;
device const char * src0_ptr = src0 + i03*nb03 + i02*nb02 + i01*nb01;
device char * dst_ptr = dst + i3*nb3 + i2*nb2 + i1*nb1 ;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int i00 = i0 % ne00;
*((device T *)(dst_ptr + i0*nb0)) = *((device T *)(src0_ptr + i00*nb00));
}
}
typedef decltype(kernel_repeat) kernel_repeat_t;
template [[host_name("kernel_repeat_f32")]] kernel kernel_repeat_t kernel_repeat;
template [[host_name("kernel_repeat_f16")]] kernel kernel_repeat_t kernel_repeat;
template [[host_name("kernel_repeat_i32")]] kernel kernel_repeat_t kernel_repeat;
template [[host_name("kernel_repeat_i16")]] kernel kernel_repeat_t kernel_repeat;
// assumption: src1 is a row
// broadcast src1 into src0
kernel void kernel_add_row(
device const float4 * src0,
device const float4 * src1,
device float4 * dst,
constant uint64_t & nb [[buffer(28)]],
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] + src1[tpig % nb];
}
kernel void kernel_sub_row(
device const float4 * src0,
device const float4 * src1,
device float4 * dst,
constant uint64_t & nb [[buffer(28)]],
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] - src1[tpig % nb];
}
kernel void kernel_mul_row(
device const float4 * src0,
device const float4 * src1,
device float4 * dst,
constant uint64_t & nb [[buffer(28)]],
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] * src1[tpig % nb];
}
kernel void kernel_div_row(
device const float4 * src0,
device const float4 * src1,
device float4 * dst,
constant uint64_t & nb [[buffer(28)]],
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] / src1[tpig % nb];
}
kernel void kernel_scale(
device const float * src0,
device float * dst,
constant float & scale,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] * scale;
}
kernel void kernel_scale_4(
device const float4 * src0,
device float4 * dst,
constant float & scale,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] * scale;
}
kernel void kernel_clamp(
device const float * src0,
device float * dst,
constant float & min,
constant float & max,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] < min ? min : (src0[tpig] > max ? max : src0[tpig]);
}
kernel void kernel_relu(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = max(0.0f, src0[tpig]);
}
kernel void kernel_sigmoid(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = 1.0f / (1.0f + exp(-src0[tpig]));
}
kernel void kernel_tanh(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
device const float & x = src0[tpig];
dst[tpig] = precise::tanh(x);
}
constant float GELU_COEF_A = 0.044715f;
constant float GELU_QUICK_COEF = -1.702f;
constant float SQRT_2_OVER_PI = 0.79788456080286535587989211986876f;
kernel void kernel_gelu(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
device const float & x = src0[tpig];
dst[tpig] = 0.5f*x*(1.0f + precise::tanh(SQRT_2_OVER_PI*x*(1.0f + GELU_COEF_A*x*x)));
}
kernel void kernel_gelu_4(
device const float4 * src0,
device float4 * dst,
uint tpig[[thread_position_in_grid]]) {
device const float4 & x = src0[tpig];
// BEWARE !!!
// Simply using "tanh" instead of "precise::tanh" will sometimes results in NaNs!
// This was observed with Falcon 7B and 40B models
//
dst[tpig] = 0.5f*x*(1.0f + precise::tanh(SQRT_2_OVER_PI*x*(1.0f + GELU_COEF_A*x*x)));
}
kernel void kernel_gelu_quick(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
device const float & x = src0[tpig];
dst[tpig] = x*(1.0f/(1.0f+exp(GELU_QUICK_COEF*x)));
}
kernel void kernel_gelu_quick_4(
device const float4 * src0,
device float4 * dst,
uint tpig[[thread_position_in_grid]]) {
device const float4 & x = src0[tpig];
dst[tpig] = x*(1.0f/(1.0f+exp(GELU_QUICK_COEF*x)));
}
kernel void kernel_silu(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
device const float & x = src0[tpig];
dst[tpig] = x / (1.0f + exp(-x));
}
kernel void kernel_silu_4(
device const float4 * src0,
device float4 * dst,
uint tpig[[thread_position_in_grid]]) {
device const float4 & x = src0[tpig];
dst[tpig] = x / (1.0f + exp(-x));
}
kernel void kernel_sqr(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] * src0[tpig];
}
kernel void kernel_sqrt(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = sqrt(src0[tpig]);
}
kernel void kernel_sin(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = sin(src0[tpig]);
}
kernel void kernel_cos(
device const float * src0,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = cos(src0[tpig]);
}
kernel void kernel_sum_rows(
device const float * src0,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tpig[[thread_position_in_grid]]) {
int64_t i3 = tpig.z;
int64_t i2 = tpig.y;
int64_t i1 = tpig.x;
if (i3 >= ne03 || i2 >= ne02 || i1 >= ne01) {
return;
}
device const float * src_row = (device const float *) ((device const char *) src0 + i1*nb01 + i2*nb02 + i3*nb03);
device float * dst_row = (device float *) ((device char *) dst + i1*nb1 + i2*nb2 + i3*nb3);
float row_sum = 0;
for (int64_t i0 = 0; i0 < ne00; i0++) {
row_sum += src_row[i0];
}
dst_row[0] = row_sum;
}
template
kernel void kernel_soft_max(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant float & scale,
constant float & max_bias,
constant float & m0,
constant float & m1,
constant uint32_t & n_head_log2,
threadgroup float * buf [[threadgroup(0)]],
uint tgpig[[threadgroup_position_in_grid]],
uint tpitg[[thread_position_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]],
uint tiisg[[thread_index_in_simdgroup]],
uint ntg[[threads_per_threadgroup]]) {
const int64_t i03 = (tgpig) / (ne02*ne01);
const int64_t i02 = (tgpig - i03*ne02*ne01) / ne01;
const int64_t i01 = (tgpig - i03*ne02*ne01 - i02*ne01);
device const float * psrc0 = (device const float *) src0 + (i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00);
device const T * pmask = src1 != src0 ? (device const T *) src1 + i01*ne00 : nullptr;
device float * pdst = (device float *) dst + (i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00);
float slope = 1.0f;
// ALiBi
if (max_bias > 0.0f) {
const int64_t h = i02;
const float base = h < n_head_log2 ? m0 : m1;
const int exp = h < n_head_log2 ? h + 1 : 2*(h - n_head_log2) + 1;
slope = pow(base, exp);
}
// parallel max
float lmax = -INFINITY;
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
lmax = MAX(lmax, psrc0[i00]*scale + (pmask ? slope*pmask[i00] : 0.0f));
}
// find the max value in the block
float max_val = simd_max(lmax);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = -INFINITY;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = max_val;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
max_val = buf[tiisg];
max_val = simd_max(max_val);
}
// parallel sum
float lsum = 0.0f;
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
const float exp_psrc0 = exp((psrc0[i00]*scale + (pmask ? slope*pmask[i00] : 0.0f)) - max_val);
lsum += exp_psrc0;
pdst[i00] = exp_psrc0;
}
// This barrier fixes a failing test
// ref: https://github.com/ggerganov/ggml/pull/621#discussion_r1425156335
threadgroup_barrier(mem_flags::mem_none);
float sum = simd_sum(lsum);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = 0.0f;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = sum;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
sum = buf[tiisg];
sum = simd_sum(sum);
}
const float inv_sum = 1.0f/sum;
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
pdst[i00] *= inv_sum;
}
}
template
kernel void kernel_soft_max_4(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant float & scale,
constant float & max_bias,
constant float & m0,
constant float & m1,
constant uint32_t & n_head_log2,
threadgroup float * buf [[threadgroup(0)]],
uint tgpig[[threadgroup_position_in_grid]],
uint tpitg[[thread_position_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]],
uint tiisg[[thread_index_in_simdgroup]],
uint ntg[[threads_per_threadgroup]]) {
const int64_t i03 = (tgpig) / (ne02*ne01);
const int64_t i02 = (tgpig - i03*ne02*ne01) / ne01;
const int64_t i01 = (tgpig - i03*ne02*ne01 - i02*ne01);
device const float4 * psrc4 = (device const float4 *) src0 + (i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00)/4;
device const T * pmask = src1 != src0 ? (device const T *) src1 + i01*ne00/4 : nullptr;
device float4 * pdst4 = (device float4 *) dst + (i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00)/4;
float slope = 1.0f;
if (max_bias > 0.0f) {
const int64_t h = i02;
const float base = h < n_head_log2 ? m0 : m1;
const int exp = h < n_head_log2 ? h + 1 : 2*(h - n_head_log2) + 1;
slope = pow(base, exp);
}
// parallel max
float4 lmax4 = -INFINITY;
for (int i00 = tpitg; i00 < ne00/4; i00 += ntg) {
lmax4 = fmax(lmax4, psrc4[i00]*scale + (float4)((pmask ? slope*pmask[i00] : 0.0f)));
}
const float lmax = MAX(MAX(lmax4[0], lmax4[1]), MAX(lmax4[2], lmax4[3]));
float max_val = simd_max(lmax);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = -INFINITY;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = max_val;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
max_val = buf[tiisg];
max_val = simd_max(max_val);
}
// parallel sum
float4 lsum4 = 0.0f;
for (int i00 = tpitg; i00 < ne00/4; i00 += ntg) {
const float4 exp_psrc4 = exp((psrc4[i00]*scale + (float4)((pmask ? slope*pmask[i00] : 0.0f))) - max_val);
lsum4 += exp_psrc4;
pdst4[i00] = exp_psrc4;
}
const float lsum = lsum4[0] + lsum4[1] + lsum4[2] + lsum4[3];
// This barrier fixes a failing test
// ref: https://github.com/ggerganov/ggml/pull/621#discussion_r1425156335
threadgroup_barrier(mem_flags::mem_none);
float sum = simd_sum(lsum);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = 0.0f;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = sum;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
sum = buf[tiisg];
sum = simd_sum(sum);
}
const float inv_sum = 1.0f/sum;
for (int i00 = tpitg; i00 < ne00/4; i00 += ntg) {
pdst4[i00] *= inv_sum;
}
}
typedef decltype(kernel_soft_max) kernel_soft_max_t;
typedef decltype(kernel_soft_max_4) kernel_soft_max_4_t;
template [[host_name("kernel_soft_max_f16")]] kernel kernel_soft_max_t kernel_soft_max;
template [[host_name("kernel_soft_max_f32")]] kernel kernel_soft_max_t kernel_soft_max;
template [[host_name("kernel_soft_max_f16_4")]] kernel kernel_soft_max_4_t kernel_soft_max_4;
template [[host_name("kernel_soft_max_f32_4")]] kernel kernel_soft_max_4_t kernel_soft_max_4;
kernel void kernel_diag_mask_inf(
device const float * src0,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int & n_past,
uint3 tpig[[thread_position_in_grid]]) {
const int64_t i02 = tpig[2];
const int64_t i01 = tpig[1];
const int64_t i00 = tpig[0];
if (i00 > n_past + i01) {
dst[i02*ne01*ne00 + i01*ne00 + i00] = -INFINITY;
} else {
dst[i02*ne01*ne00 + i01*ne00 + i00] = src0[i02*ne01*ne00 + i01*ne00 + i00];
}
}
kernel void kernel_diag_mask_inf_8(
device const float4 * src0,
device float4 * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int & n_past,
uint3 tpig[[thread_position_in_grid]]) {
const int64_t i = 2*tpig[0];
dst[i+0] = src0[i+0];
dst[i+1] = src0[i+1];
int64_t i4 = 4*i;
const int64_t i02 = i4/(ne00*ne01); i4 -= i02*ne00*ne01;
const int64_t i01 = i4/(ne00); i4 -= i01*ne00;
const int64_t i00 = i4;
for (int k = 3; k >= 0; --k) {
if (i00 + 4 + k <= n_past + i01) {
break;
}
dst[i+1][k] = -INFINITY;
if (i00 + k > n_past + i01) {
dst[i][k] = -INFINITY;
}
}
}
// ref: ggml.c:ggml_compute_forward_ssm_conv_f32
// TODO: optimize
kernel void kernel_ssm_conv_f32(
device const void * src0,
device const void * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t ir = tgpig.x;
const int64_t i2 = tgpig.y;
const int64_t i3 = tgpig.z;
const int64_t nc = ne10;
const int64_t ncs = ne00;
const int64_t nr = ne01;
const int64_t n_t = ne1;
const int64_t n_s = ne2;
device const float * s = (device const float *) ((device const char *) src0 + ir*nb01 + i2*nb00 + i3*nb02);
device const float * c = (device const float *) ((device const char *) src1 + ir*nb11);
device float * x = (device float *) ((device char *) dst + ir*nb0 + i2*nb1 + i3*nb2);
float sumf = 0.0f;
for (int64_t i0 = 0; i0 < nc; ++i0) {
sumf += s[i0] * c[i0];
}
x[0] = sumf;
}
// ref: ggml.c:ggml_compute_forward_ssm_scan_f32
// TODO: optimize
kernel void kernel_ssm_scan_f32(
device const void * src0,
device const void * src1,
device const void * src2,
device const void * src3,
device const void * src4,
device const void * src5,
device float * dst,
constant int64_t & d_state,
constant int64_t & d_inner,
constant int64_t & n_seq_tokens,
constant int64_t & n_seqs,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant uint64_t & nb20,
constant uint64_t & nb21,
constant uint64_t & nb22,
constant uint64_t & nb30,
constant uint64_t & nb31,
constant uint64_t & nb40,
constant uint64_t & nb41,
constant uint64_t & nb42,
constant uint64_t & nb50,
constant uint64_t & nb51,
constant uint64_t & nb52,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t ir = tgpig.x;
const int64_t i3 = tgpig.y;
const int64_t nc = d_state;
const int64_t nr = d_inner;
const int64_t n_t = n_seq_tokens;
const int64_t n_s = n_seqs;
for (int64_t i2 = 0; i2 < n_t; ++i2) {
device const float * s0 = (device const float *) ((device const char *) src0 + ir*nb01 + i3*nb02);
device const float * x = (device const float *) ((device const char *) src1 + ir*nb10 + i2*nb11 + i3*nb12);
device const float * dt = (device const float *) ((device const char *) src2 + ir*nb20 + i2*nb21 + i3*nb22);
device const float * A = (device const float *) ((device const char *) src3 + ir*nb31);
device const float * B = (device const float *) ((device const char *) src4 + i2*nb41 + i3*nb42);
device const float * C = (device const float *) ((device const char *) src5 + i2*nb51 + i3*nb52);
device float * y = (device float *) ((device char *) dst + ir*nb10 + i2*nb11 + i3*nb12); // TODO: do not use src1 strides
device float * s = (device float *) ((device char *) dst + ir*nb01 + i3*nb02 + nb13);
if (i2 > 0) {
s0 = s;
}
// i1 == 0
float dt_soft_plus = dt[0] <= 20.0f ? log(1.0f + exp(dt[0])) : dt[0];
float x_dt = x[0] * dt_soft_plus;
float sumf = 0.0f;
for (int64_t i0 = 0; i0 < nc; ++i0) {
int64_t i = i0;
float state = (s0[i] * exp(dt_soft_plus * A[i])) + (B[i0] * x_dt);
sumf += state * C[i0];
s[i] = state;
}
y[0] = sumf;
}
}
kernel void kernel_norm(
device const void * src0,
device float * dst,
constant int64_t & ne00,
constant uint64_t & nb01,
constant float & eps,
threadgroup float * sum [[threadgroup(0)]],
uint tgpig[[threadgroup_position_in_grid]],
uint tpitg[[thread_position_in_threadgroup]],
uint ntg[[threads_per_threadgroup]]) {
device const float * x = (device const float *) ((device const char *) src0 + tgpig*nb01);
// MEAN
// parallel sum
sum[tpitg] = 0.0f;
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
sum[tpitg] += x[i00];
}
// reduce
threadgroup_barrier(mem_flags::mem_threadgroup);
for (uint i = ntg/2; i > 0; i /= 2) {
if (tpitg < i) {
sum[tpitg] += sum[tpitg + i];
}
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const float mean = sum[0] / ne00;
// recenter and VARIANCE
threadgroup_barrier(mem_flags::mem_threadgroup);
device float * y = dst + tgpig*ne00;
sum[tpitg] = 0.0f;
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
y[i00] = x[i00] - mean;
sum[tpitg] += y[i00] * y[i00];
}
// reduce
threadgroup_barrier(mem_flags::mem_threadgroup);
for (uint i = ntg/2; i > 0; i /= 2) {
if (tpitg < i) {
sum[tpitg] += sum[tpitg + i];
}
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const float variance = sum[0] / ne00;
const float scale = 1.0f/sqrt(variance + eps);
for (int i00 = tpitg; i00 < ne00; i00 += ntg) {
y[i00] = y[i00] * scale;
}
}
kernel void kernel_rms_norm(
device const void * src0,
device float * dst,
constant int64_t & ne00,
constant uint64_t & nb01,
constant float & eps,
threadgroup float * buf [[threadgroup(0)]],
uint tgpig[[threadgroup_position_in_grid]],
uint tpitg[[thread_position_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]],
uint tiisg[[thread_index_in_simdgroup]],
uint ntg[[threads_per_threadgroup]]) {
device const float4 * x = (device const float4 *) ((device const char *) src0 + tgpig*nb01);
float4 sumf = 0;
float all_sum = 0;
// parallel sum
for (int i00 = tpitg; i00 < ne00/4; i00 += ntg) {
sumf += x[i00] * x[i00];
}
all_sum = sumf[0] + sumf[1] + sumf[2] + sumf[3];
all_sum = simd_sum(all_sum);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = 0.0f;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = all_sum;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
all_sum = buf[tiisg];
all_sum = simd_sum(all_sum);
}
const float mean = all_sum/ne00;
const float scale = 1.0f/sqrt(mean + eps);
device float4 * y = (device float4 *) (dst + tgpig*ne00);
for (int i00 = tpitg; i00 < ne00/4; i00 += ntg) {
y[i00] = x[i00] * scale;
}
}
kernel void kernel_group_norm(
device const float * src0,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int32_t & n_groups,
constant float & eps,
threadgroup float * buf [[threadgroup(0)]],
uint tgpig[[threadgroup_position_in_grid]],
uint tpitg[[thread_position_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]],
uint tiisg[[thread_index_in_simdgroup]],
uint ntg[[threads_per_threadgroup]]) {
const int64_t ne = ne00*ne01*ne02;
const int64_t gs = ne00*ne01*((ne02 + n_groups - 1) / n_groups);
int start = tgpig * gs;
int end = start + gs;
start += tpitg;
if (end >= ne) {
end = ne;
}
float tmp = 0.0f; // partial sum for thread in warp
for (int j = start; j < end; j += ntg) {
tmp += src0[j];
}
threadgroup_barrier(mem_flags::mem_threadgroup);
tmp = simd_sum(tmp);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = 0.0f;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = tmp;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
tmp = buf[tiisg];
tmp = simd_sum(tmp);
}
const float mean = tmp / gs;
tmp = 0.0f;
for (int j = start; j < end; j += ntg) {
float xi = src0[j] - mean;
dst[j] = xi;
tmp += xi * xi;
}
tmp = simd_sum(tmp);
if (ntg > N_SIMDWIDTH) {
if (sgitg == 0) {
buf[tiisg] = 0.0f;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
if (tiisg == 0) {
buf[sgitg] = tmp;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
tmp = buf[tiisg];
tmp = simd_sum(tmp);
}
const float variance = tmp / gs;
const float scale = 1.0f/sqrt(variance + eps);
for (int j = start; j < end; j += ntg) {
dst[j] *= scale;
}
}
// function for calculate inner product between half a q4_0 block and 16 floats (yl), sumy is SUM(yl[i])
// il indicates where the q4 quants begin (0 or QK4_0/4)
// we assume that the yl's have been multiplied with the appropriate scale factor
// that corresponds to the missing bit shifts (1, 1/16, 1/256, 1/4096)
inline float block_q_n_dot_y(device const block_q4_0 * qb_curr, float sumy, thread float * yl, int il) {
float d = qb_curr->d;
float2 acc = 0.f;
device const uint16_t * qs = ((device const uint16_t *)qb_curr + 1 + il/2);
for (int i = 0; i < 8; i+=2) {
acc[0] += yl[i + 0] * (qs[i / 2] & 0x000F)
+ yl[i + 1] * (qs[i / 2] & 0x0F00);
acc[1] += yl[i + 8] * (qs[i / 2] & 0x00F0)
+ yl[i + 9] * (qs[i / 2] & 0xF000);
}
return d * (sumy * -8.f + acc[0] + acc[1]);
}
// function for calculate inner product between half a q4_1 block and 16 floats (yl), sumy is SUM(yl[i])
// il indicates where the q4 quants begin (0 or QK4_0/4)
// we assume that the yl's have been multiplied with the appropriate scale factor
// that corresponds to the missing bit shifts (1, 1/16, 1/256, 1/4096)
inline float block_q_n_dot_y(device const block_q4_1 * qb_curr, float sumy, thread float * yl, int il) {
float d = qb_curr->d;
float m = qb_curr->m;
float2 acc = 0.f;
device const uint16_t * qs = ((device const uint16_t *)qb_curr + 2 + il/2);
for (int i = 0; i < 8; i+=2) {
acc[0] += yl[i + 0] * (qs[i / 2] & 0x000F)
+ yl[i + 1] * (qs[i / 2] & 0x0F00);
acc[1] += yl[i + 8] * (qs[i / 2] & 0x00F0)
+ yl[i + 9] * (qs[i / 2] & 0xF000);
}
return d * (acc[0] + acc[1]) + sumy * m;
}
// function for calculate inner product between half a q5_0 block and 16 floats (yl), sumy is SUM(yl[i])
// il indicates where the q5 quants begin (0 or QK5_0/4)
// we assume that the yl's have been multiplied with the appropriate scale factor
// that corresponds to the missing bit shifts (1, 1/16, 1/256, 1/4096)
inline float block_q_n_dot_y(device const block_q5_0 * qb_curr, float sumy, thread float * yl, int il) {
float d = qb_curr->d;
float2 acc = 0.f;
device const uint16_t * qs = ((device const uint16_t *)qb_curr + 3 + il/2);
const uint32_t qh = *((device const uint32_t *)qb_curr->qh);
for (int i = 0; i < 8; i+=2) {
acc[0] += yl[i + 0] * ((qs[i / 2] & 0x000F) | ((qh >> (i+0+il ) << 4 ) & 0x00010))
+ yl[i + 1] * ((qs[i / 2] & 0x0F00) | ((qh >> (i+1+il ) << 12) & 0x01000));
acc[1] += yl[i + 8] * ((qs[i / 2] & 0x00F0) | ((qh >> (i+0+il+QK5_0/2) << 8 ) & 0x00100))
+ yl[i + 9] * ((qs[i / 2] & 0xF000) | ((qh >> (i+1+il+QK5_0/2) << 16) & 0x10000));
}
return d * (sumy * -16.f + acc[0] + acc[1]);
}
// function for calculate inner product between half a q5_1 block and 16 floats (yl), sumy is SUM(yl[i])
// il indicates where the q5 quants begin (0 or QK5_1/4)
// we assume that the yl's have been multiplied with the appropriate scale factor
// that corresponds to the missing bit shifts (1, 1/16, 1/256, 1/4096)
inline float block_q_n_dot_y(device const block_q5_1 * qb_curr, float sumy, thread float * yl, int il) {
float d = qb_curr->d;
float m = qb_curr->m;
float2 acc = 0.f;
device const uint16_t * qs = ((device const uint16_t *)qb_curr + 4 + il/2);
const uint32_t qh = *((device const uint32_t *)qb_curr->qh);
for (int i = 0; i < 8; i+=2) {
acc[0] += yl[i + 0] * ((qs[i / 2] & 0x000F) | ((qh >> (i+0+il ) << 4 ) & 0x00010))
+ yl[i + 1] * ((qs[i / 2] & 0x0F00) | ((qh >> (i+1+il ) << 12) & 0x01000));
acc[1] += yl[i + 8] * ((qs[i / 2] & 0x00F0) | ((qh >> (i+0+il+QK5_0/2) << 8 ) & 0x00100))
+ yl[i + 9] * ((qs[i / 2] & 0xF000) | ((qh >> (i+1+il+QK5_0/2) << 16) & 0x10000));
}
return d * (acc[0] + acc[1]) + sumy * m;
}
// putting them in the kernel cause a significant performance penalty
#define N_DST 4 // each SIMD group works on 4 rows
#define N_SIMDGROUP 2 // number of SIMD groups in a thread group
//Note: This is a template, but strictly speaking it only applies to
// quantizations where the block size is 32. It also does not
// guard against the number of rows not being divisible by
// N_DST, so this is another explicit assumption of the implementation.
template
void mul_vec_q_n_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig, uint tiisg, uint sgitg) {
const int nb = ne00/QK4_0;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * nsg + sgitg) * nr;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = first_row * nb + (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q_type * x = (device const block_q_type *) src0 + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[16]; // src1 vector cache
float sumf[nr] = {0.f};
const int ix = (tiisg/2);
const int il = (tiisg%2)*8;
device const float * yb = y + ix * QK4_0 + il;
// each thread in a SIMD group deals with half a block.
for (int ib = ix; ib < nb; ib += nw/2) {
float sumy = 0;
for (int i = 0; i < 8; i += 2) {
sumy += yb[i] + yb[i+1];
yl[i+0] = yb[i+ 0];
yl[i+1] = yb[i+ 1]/256.f;
sumy += yb[i+16] + yb[i+17];
yl[i+8] = yb[i+16]/16.f;
yl[i+9] = yb[i+17]/4096.f;
}
for (int row = 0; row < nr; row++) {
sumf[row] += block_q_n_dot_y(x+ib+row*nb, sumy, yl, il);
}
yb += QK4_0 * 16;
}
for (int row = 0; row < nr; ++row) {
const float tot = simd_sum(sumf[row]);
if (tiisg == 0 && first_row + row < ne01) {
dst[im*ne0*ne1 + r1*ne0 + first_row + row] = tot;
}
}
}
kernel void kernel_mul_mv_q4_0_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
mul_vec_q_n_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);
}
kernel void kernel_mul_mv_q4_1_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
mul_vec_q_n_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);
}
kernel void kernel_mul_mv_q5_0_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
mul_vec_q_n_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);
}
kernel void kernel_mul_mv_q5_1_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
mul_vec_q_n_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);
}
#define NB_Q8_0 8
void kernel_mul_mv_q8_0_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nr = N_DST;
const int nsg = N_SIMDGROUP;
const int nw = N_SIMDWIDTH;
const int nb = ne00/QK8_0;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * nsg + sgitg) * nr;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = first_row * nb + (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q8_0 * x = (device const block_q8_0 *) src0 + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[NB_Q8_0];
float sumf[nr]={0.f};
const int ix = tiisg/4;
const int il = tiisg%4;
device const float * yb = y + ix * QK8_0 + NB_Q8_0*il;
// each thread in a SIMD group deals with NB_Q8_0 quants at a time
for (int ib = ix; ib < nb; ib += nw/4) {
for (int i = 0; i < NB_Q8_0; ++i) {
yl[i] = yb[i];
}
for (int row = 0; row < nr; row++) {
device const int8_t * qs = x[ib+row*nb].qs + NB_Q8_0*il;
float sumq = 0.f;
for (int iq = 0; iq < NB_Q8_0; ++iq) {
sumq += qs[iq] * yl[iq];
}
sumf[row] += sumq*x[ib+row*nb].d;
}
yb += NB_Q8_0 * nw;
}
for (int row = 0; row < nr; ++row) {
const float tot = simd_sum(sumf[row]);
if (tiisg == 0 && first_row + row < ne01) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = tot;
}
}
}
[[host_name("kernel_mul_mv_q8_0_f32")]]
kernel void kernel_mul_mv_q8_0_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q8_0_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);
}
#define N_MV_T_T 4
template
void kernel_mul_mv_impl(
device const char * src0,
device const char * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
uint64_t nb00,
uint64_t nb01,
uint64_t nb02,
int64_t ne10,
int64_t ne11,
int64_t ne12,
uint64_t nb10,
uint64_t nb11,
uint64_t nb12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
uint3 tgpig,
uint tiisg) {
const int64_t r0 = tgpig.x;
const int64_t rb = tgpig.y*N_MV_T_T;
const int64_t im = tgpig.z;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = r0*nb01 + (i12/r2)*nb02 + (i13/r3)*nb02*ne02;
device const T0 * x = (device const T0 *) (src0 + offset0);
if (ne00 < 128) {
for (int row = 0; row < N_MV_T_T; ++row) {
int r1 = rb + row;
if (r1 >= ne11) {
break;
}
device const T1 * y = (device const T1 *) (src1 + r1*nb11 + im*nb12);
float sumf = 0;
for (int i = tiisg; i < ne00; i += 32) {
sumf += (T0) x[i] * (T1) y[i];
}
float all_sum = simd_sum(sumf);
if (tiisg == 0) {
dst[im*ne1*ne0 + r1*ne0 + r0] = all_sum;
}
}
} else {
device const T04 * x4 = (device const T04 *) x;
for (int row = 0; row < N_MV_T_T; ++row) {
int r1 = rb + row;
if (r1 >= ne11) {
break;
}
device const T1 * y = (device const T1 *) (src1 + r1*nb11 + im*nb12);
device const T14 * y4 = (device const T14 *) y;
float sumf = 0;
for (int i = tiisg; i < ne00/4; i += 32) {
for (int k = 0; k < 4; ++k) sumf += (float) (x4[i][k] * y4[i][k]);
}
float all_sum = simd_sum(sumf);
if (tiisg == 0) {
for (int i = 4*(ne00/4); i < ne00; ++i) all_sum += (float) (x[i] * y[i]);
dst[im*ne1*ne0 + r1*ne0 + r0] = all_sum;
}
}
}
}
template
kernel void kernel_mul_mv(
device const char * src0,
device const char * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]]) {
kernel_mul_mv_impl(
src0,
src1,
dst,
ne00,
ne01,
ne02,
nb00,
nb01,
nb02,
ne10,
ne11,
ne12,
nb10,
nb11,
nb12,
ne0,
ne1,
r2,
r3,
tgpig,
tiisg);
}
typedef decltype(kernel_mul_mv) mul_mv_t;
template [[host_name("kernel_mul_mv_f32_f32")]] kernel mul_mv_t kernel_mul_mv;
template [[host_name("kernel_mul_mv_f16_f32")]] kernel mul_mv_t kernel_mul_mv;
template [[host_name("kernel_mul_mv_f16_f16")]] kernel mul_mv_t kernel_mul_mv;
template
kernel void kernel_mul_mv_1row(
device const char * src0,
device const char * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]]) {
const int64_t r0 = tgpig.x;
const int64_t r1 = tgpig.y;
const int64_t im = tgpig.z;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = r0*nb01 + (i12/r2)*nb02 + (i13/r3)*nb02*ne02;
device const T * x = (device const T *) (src0 + offset0);
device const float * y = (device const float *) (src1 + r1*nb11 + im*nb12);
float sumf = 0;
if (ne00 < 128) {
for (int i = tiisg; i < ne00; i += 32) {
sumf += (float) x[i] * (float) y[i];
}
float all_sum = simd_sum(sumf);
if (tiisg == 0) {
dst[im*ne1*ne0 + r1*ne0 + r0] = all_sum;
}
} else {
device const T4 * x4 = (device const T4 *) x;
device const float4 * y4 = (device const float4 *) y;
for (int i = tiisg; i < ne00/4; i += 32) {
for (int k = 0; k < 4; ++k) sumf += (float) (x4[i][k] * y4[i][k]);
}
float all_sum = simd_sum(sumf);
if (tiisg == 0) {
for (int i = 4*(ne00/4); i < ne00; ++i) all_sum += (float) (x[i] * y[i]);
dst[im*ne1*ne0 + r1*ne0 + r0] = all_sum;
}
}
}
typedef decltype(kernel_mul_mv_1row) mul_mv_1row_t;
template [[host_name("kernel_mul_mv_f16_f32_1row")]] kernel mul_mv_1row_t kernel_mul_mv_1row;
// Assumes row size (ne00) is a multiple of 4
template
kernel void kernel_mul_mv_l4(
device const char * src0,
device const char * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]]) {
const int nrows = ne11;
const int64_t r0 = tgpig.x;
const int64_t im = tgpig.z;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = r0*nb01 + (i12/r2)*nb02 + (i13/r3)*nb02*ne02;
device const T4 * x4 = (device const T4 *) (src0 + offset0);
for (int r1 = 0; r1 < nrows; ++r1) {
device const float4 * y4 = (device const float4 *) (src1 + r1*nb11 + im*nb12);
float sumf = 0;
for (int i = tiisg; i < ne00/4; i += 32) {
for (int k = 0; k < 4; ++k) sumf += (float) (x4[i][k] * y4[i][k]);
}
float all_sum = simd_sum(sumf);
if (tiisg == 0) {
dst[im*ne1*ne0 + r1*ne0 + r0] = all_sum;
}
}
}
typedef decltype(kernel_mul_mv_l4) mul_mv_l4_t;
template [[host_name("kernel_mul_mv_f16_f32_l4")]] kernel mul_mv_l4_t kernel_mul_mv_l4;
static float rope_yarn_ramp(const float low, const float high, const int i0) {
const float y = (i0 / 2 - low) / max(0.001f, high - low);
return 1.0f - min(1.0f, max(0.0f, y));
}
// YaRN algorithm based on LlamaYaRNScaledRotaryEmbedding.py from https://github.com/jquesnelle/yarn
// MIT licensed. Copyright (c) 2023 Jeffrey Quesnelle and Bowen Peng.
static void rope_yarn(
float theta_extrap, float freq_scale, float corr_dims[2], int64_t i0, float ext_factor, float mscale,
thread float * cos_theta, thread float * sin_theta) {
// Get n-d rotational scaling corrected for extrapolation
float theta_interp = freq_scale * theta_extrap;
float theta = theta_interp;
if (ext_factor != 0.0f) {
float ramp_mix = rope_yarn_ramp(corr_dims[0], corr_dims[1], i0) * ext_factor;
theta = theta_interp * (1 - ramp_mix) + theta_extrap * ramp_mix;
// Get n-d magnitude scaling corrected for interpolation
mscale *= 1.0f + 0.1f * log(1.0f / freq_scale);
}
*cos_theta = cos(theta) * mscale;
*sin_theta = sin(theta) * mscale;
}
// Apparently solving `n_rot = 2pi * x * base^((2 * max_pos_emb) / n_dims)` for x, we get
// `corr_fac(n_rot) = n_dims * log(max_pos_emb / (n_rot * 2pi)) / (2 * log(base))`
static float rope_yarn_corr_factor(int n_dims, int n_ctx_orig, float n_rot, float base) {
return n_dims * log(n_ctx_orig / (n_rot * 2 * M_PI_F)) / (2 * log(base));
}
static void rope_yarn_corr_dims(
int n_dims, int n_ctx_orig, float freq_base, float beta_fast, float beta_slow, float dims[2]
) {
// start and end correction dims
dims[0] = max(0.0f, floor(rope_yarn_corr_factor(n_dims, n_ctx_orig, beta_fast, freq_base)));
dims[1] = min(n_dims - 1.0f, ceil(rope_yarn_corr_factor(n_dims, n_ctx_orig, beta_slow, freq_base)));
}
template
kernel void kernel_rope_norm(
device const void * src0,
device const int32_t * src1,
device const float * src2,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant int & n_past,
constant int & n_dims,
constant int & n_ctx_orig,
constant float & freq_base,
constant float & freq_scale,
constant float & ext_factor,
constant float & attn_factor,
constant float & beta_fast,
constant float & beta_slow,
uint tiitg[[thread_index_in_threadgroup]],
uint3 tptg[[threads_per_threadgroup]],
uint3 tgpig[[threadgroup_position_in_grid]]) {
const int64_t i3 = tgpig[2];
const int64_t i2 = tgpig[1];
const int64_t i1 = tgpig[0];
float corr_dims[2];
rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, corr_dims);
device const int32_t * pos = src1;
const float theta_base = (float) pos[i2];
const float inv_ndims = -1.f/n_dims;
float cos_theta;
float sin_theta;
for (int64_t i0 = 2*tiitg; i0 < ne0; i0 += 2*tptg.x) {
if (i0 < n_dims) {
const int64_t ic = i0/2;
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor, &cos_theta, &sin_theta);
device const T * const src = (device T *)((device char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
device T * dst_data = (device T *)((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
const float x0 = src[0];
const float x1 = src[1];
dst_data[0] = x0*cos_theta - x1*sin_theta;
dst_data[1] = x0*sin_theta + x1*cos_theta;
} else {
device const T * const src = (device T *)((device char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
device T * dst_data = (device T *)((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
dst_data[0] = src[0];
dst_data[1] = src[1];
}
}
}
template
kernel void kernel_rope_neox(
device const void * src0,
device const int32_t * src1,
device const float * src2,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant int & n_past,
constant int & n_dims,
constant int & n_ctx_orig,
constant float & freq_base,
constant float & freq_scale,
constant float & ext_factor,
constant float & attn_factor,
constant float & beta_fast,
constant float & beta_slow,
uint tiitg[[thread_index_in_threadgroup]],
uint3 tptg[[threads_per_threadgroup]],
uint3 tgpig[[threadgroup_position_in_grid]]) {
const int64_t i3 = tgpig[2];
const int64_t i2 = tgpig[1];
const int64_t i1 = tgpig[0];
float corr_dims[2];
rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, corr_dims);
device const int32_t * pos = src1;
const float theta_base = (float) pos[i2];
const float inv_ndims = -1.f/n_dims;
float cos_theta;
float sin_theta;
for (int64_t i0 = 2*tiitg; i0 < ne0; i0 += 2*tptg.x) {
if (i0 < n_dims) {
const int64_t ic = i0/2;
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor, &cos_theta, &sin_theta);
device const T * const src = (device T *)((device char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + ic*nb00);
device T * dst_data = (device T *)((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + ic*nb0);
const float x0 = src[0];
const float x1 = src[n_dims/2];
dst_data[0] = x0*cos_theta - x1*sin_theta;
dst_data[n_dims/2] = x0*sin_theta + x1*cos_theta;
} else {
device const T * const src = (device T *)((device char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
device T * dst_data = (device T *)((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
dst_data[0] = src[0];
dst_data[1] = src[1];
}
}
}
typedef decltype(kernel_rope_norm) kernel_rope_norm_t;
typedef decltype(kernel_rope_neox) kernel_rope_neox_t;
template [[host_name("kernel_rope_norm_f32")]] kernel kernel_rope_norm_t kernel_rope_norm;
template [[host_name("kernel_rope_norm_f16")]] kernel kernel_rope_norm_t kernel_rope_norm;
template [[host_name("kernel_rope_neox_f32")]] kernel kernel_rope_neox_t kernel_rope_neox;
template [[host_name("kernel_rope_neox_f16")]] kernel kernel_rope_neox_t kernel_rope_neox;
typedef void (im2col_t)(
device const float * x,
device char * dst,
constant int32_t & ofs0,
constant int32_t & ofs1,
constant int32_t & IW,
constant int32_t & IH,
constant int32_t & CHW,
constant int32_t & s0,
constant int32_t & s1,
constant int32_t & p0,
constant int32_t & p1,
constant int32_t & d0,
constant int32_t & d1,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tgpg[[threadgroups_per_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]);
template
kernel void kernel_im2col(
device const float * x,
device char * dst,
constant int32_t & ofs0,
constant int32_t & ofs1,
constant int32_t & IW,
constant int32_t & IH,
constant int32_t & CHW,
constant int32_t & s0,
constant int32_t & s1,
constant int32_t & p0,
constant int32_t & p1,
constant int32_t & d0,
constant int32_t & d1,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tgpg[[threadgroups_per_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int32_t iiw = tgpig[2] * s0 + tpitg[2] * d0 - p0;
const int32_t iih = tgpig[1] * s1 + tpitg[1] * d1 - p1;
const int32_t offset_dst =
(tpitg[0] * tgpg[1] * tgpg[2] + tgpig[1] * tgpg[2] + tgpig[2]) * CHW +
(tgpig[0] * (ntg[1] * ntg[2]) + tpitg[1] * ntg[2] + tpitg[2]);
device T * pdst = (device T *) (dst);
if (iih < 0 || iih >= IH || iiw < 0 || iiw >= IW) {
pdst[offset_dst] = 0.0f;
} else {
const int32_t offset_src = tpitg[0] * ofs0 + tgpig[0] * ofs1;
pdst[offset_dst] = x[offset_src + iih * IW + iiw];
}
}
template [[host_name("kernel_im2col_f32")]] kernel im2col_t kernel_im2col;
template [[host_name("kernel_im2col_f16")]] kernel im2col_t kernel_im2col;
kernel void kernel_upscale_f32(
device const char * src0,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant float & sf0,
constant float & sf1,
constant float & sf2,
constant float & sf3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i3 = tgpig.z;
const int64_t i2 = tgpig.y;
const int64_t i1 = tgpig.x;
const int64_t i03 = i3/sf3;
const int64_t i02 = i2/sf2;
const int64_t i01 = i1/sf1;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
const int64_t i00 = i0/sf0;
device const float * src0_ptr = (device const float *) (src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
device float * dst_ptr = (device float *) (dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
dst_ptr[0] = src0_ptr[0];
}
}
kernel void kernel_pad_f32(
device const char * src0,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i3 = tgpig.z;
const int64_t i2 = tgpig.y;
const int64_t i1 = tgpig.x;
const int64_t i03 = i3;
const int64_t i02 = i2;
const int64_t i01 = i1;
device const float * src0_ptr = (device const float *) (src0 + i03*nb03 + i02*nb02 + i01*nb01);
device float * dst_ptr = (device float *) (dst + i3*nb3 + i2*nb2 + i1*nb1);
if (i1 < ne01 && i2 < ne02 && i3 < ne03) {
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
if (i0 < ne00) {
dst_ptr[i0] = src0_ptr[i0];
} else {
dst_ptr[i0] = 0.0f;
}
}
return;
}
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
dst_ptr[i0] = 0.0f;
}
}
kernel void kernel_arange_f32(
device char * dst,
constant int64_t & ne0,
constant float & start,
constant float & step,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
device float * dst_ptr = (device float *) dst;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
dst_ptr[i0] = start + step * i0;
}
}
kernel void kernel_timestep_embedding_f32(
device const char * src0,
device char * dst,
constant uint64_t & nb1,
constant int & dim,
constant int & max_period,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
int i = tgpig.x;
device float * embed_data = (device float *)(dst + i*nb1);
int half_ = dim / 2;
for (int j = tpitg.x; j < half_; j += ntg.x) {
float timestep = ((device float *)src0)[i];
float freq = (float)exp(-log((float)max_period) * j / half_);
float arg = timestep * freq;
embed_data[j ] = cos(arg);
embed_data[j + half_] = sin(arg);
}
if (dim % 2 != 0 && tpitg.x == 0) {
embed_data[dim] = 0.f;
}
}
// bitonic sort implementation following the CUDA kernels as reference
typedef void (argsort_t)(
device const float * x,
device int32_t * dst,
constant int64_t & ncols,
constant int64_t & ncols_pad,
threadgroup int32_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]]);
template
kernel void kernel_argsort_f32_i32(
device const float * x,
device int32_t * dst,
constant int64_t & ncols,
constant int64_t & ncols_pad,
threadgroup int32_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]]) {
// bitonic sort
int col = tpitg[0];
int row = tgpig[1];
if (col >= ncols_pad) return;
device const float * x_row = x + row * ncols;
threadgroup int32_t * dst_row = shared_values;
// initialize indices
dst_row[col] = col;
threadgroup_barrier(mem_flags::mem_threadgroup);
for (int k = 2; k <= ncols_pad; k *= 2) {
for (int j = k / 2; j > 0; j /= 2) {
int ixj = col ^ j;
if (ixj > col) {
if ((col & k) == 0) {
if (dst_row[col] >= ncols ||
(dst_row[ixj] < ncols && (order == GGML_SORT_ORDER_ASC ?
x_row[dst_row[col]] > x_row[dst_row[ixj]] :
x_row[dst_row[col]] < x_row[dst_row[ixj]]))
) {
SWAP(dst_row[col], dst_row[ixj]);
}
} else {
if (dst_row[ixj] >= ncols ||
(dst_row[col] < ncols && (order == GGML_SORT_ORDER_ASC ?
x_row[dst_row[col]] < x_row[dst_row[ixj]] :
x_row[dst_row[col]] > x_row[dst_row[ixj]]))
) {
SWAP(dst_row[col], dst_row[ixj]);
}
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
}
}
// copy the result to dst without the padding
if (col < ncols) {
dst[row * ncols + col] = dst_row[col];
}
}
template [[host_name("kernel_argsort_f32_i32_asc")]] kernel argsort_t kernel_argsort_f32_i32;
template [[host_name("kernel_argsort_f32_i32_desc")]] kernel argsort_t kernel_argsort_f32_i32;
kernel void kernel_leaky_relu_f32(
device const float * src0,
device float * dst,
constant float & slope,
uint tpig[[thread_position_in_grid]]) {
dst[tpig] = src0[tpig] > 0.0f ? src0[tpig] : src0[tpig] * slope;
}
typedef void (flash_attn_ext_f16_t)(
device const char * q,
device const char * k,
device const char * v,
device const char * mask,
device float * dst,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant uint64_t & nb21,
constant uint64_t & nb22,
constant uint64_t & nb23,
constant uint64_t & nb31,
constant int64_t & ne1,
constant int64_t & ne2,
constant float & scale,
constant float & max_bias,
constant float & m0,
constant float & m1,
constant uint32_t & n_head_log2,
constant float & logit_softcap,
threadgroup half * shared,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]],
ushort tiisg[[thread_index_in_simdgroup]],
ushort sgitg[[simdgroup_index_in_threadgroup]]);
// ref: https://arxiv.org/pdf/2307.08691.pdf
template // head size, queries per threadgroup, cache items per threadgroup
kernel void kernel_flash_attn_ext_f16(
device const char * q,
device const char * k,
device const char * v,
device const char * mask,
device float * dst,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant uint64_t & nb21,
constant uint64_t & nb22,
constant uint64_t & nb23,
constant uint64_t & nb31,
constant int64_t & ne1,
constant int64_t & ne2,
constant float & scale,
constant float & max_bias,
constant float & m0,
constant float & m1,
constant uint32_t & n_head_log2,
constant float & logit_softcap,
threadgroup half * shared [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]],
ushort tiisg[[thread_index_in_simdgroup]],
ushort sgitg[[simdgroup_index_in_threadgroup]]) {
const short nsg = ntg.y; // number of simdgroups
const short iq3 = tgpig[2];
const short iq2 = tgpig[1];
const short iq1 = tgpig[0]*Q;
const short D4 = D/4;
const short D8 = D/8;
//const short Q8 = Q/8;
const short NW = N_SIMDWIDTH;
const short SH = (C + Q); // shared memory per simdgroup in (half)
const short T = D + 2*nsg*SH; // shared memory size per query in (half)
const short TF = T/2; // shared memory size per query in (float)
const short T4 = T/4; // shared memory size per query in (half4)
threadgroup half * sq = (threadgroup half *) (shared + 0*D); // holds the query data
threadgroup half4 * sq4 = (threadgroup half4 *) (shared + 0*D); // same as above but in half4
threadgroup float * ss = (threadgroup float *) (shared + 2*sgitg*SH + 1*D); // scratch buffer for attention and diagonal matrix
// store the result for all queries in local memory in 8x8 matrices (the O matrix from the paper)
simdgroup_half8x8 lo[D8];
// load heads from Q to shared memory
for (short j = sgitg; j < Q; j += nsg) {
device const float4 * q4 = (device const float4 *) ((device const char *) q + ((iq1 + j)*nb01 + iq2*nb02 + iq3*nb03));
for (short i = tiisg; i < D4; i += NW) {
if (iq1 + j < ne01) {
sq4[j*T4 + i] = (half4) q4[i];
} else {
sq4[j*T4 + i] = 0.0h;
}
}
}
// zero out lo
for (short i = 0; i < D8; ++i) {
lo[i] = make_filled_simdgroup_matrix(0.0h);
}
// zero out shared memory SH
for (short j = 0; j < Q; ++j) {
for (short i = tiisg; i < SH; i += NW) {
ss[j*TF + i] = 0.0f;
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
{
float S[Q] = { [0 ... Q-1] = 0.0h };
float M[Q] = { [0 ... Q-1] = -FLT_MAX/2 };
// assume K and V are same shape
const short ne22 = ne12;
const short ne23 = ne13;
// broadcast
const short rk2 = ne02/ne12;
const short rk3 = ne03/ne13;
const short rv2 = ne02/ne22;
const short rv3 = ne03/ne23;
// k indices
const short ik2 = iq2/rk2;
const short ik3 = iq3/rk3;
// v indices
const short iv2 = iq2/rv2;
const short iv3 = iq3/rv3;
// load the queries from shared memory into local memory
simdgroup_half8x8 mq[D8];
for (short i = 0; i < D8; ++i) {
simdgroup_load(mq[i], sq + i*8, T);
}
// pointer to the mask
device const half * mp = (device const half *) (mask + iq1*nb31);
float slope = 1.0f;
// ALiBi
if (max_bias > 0.0f) {
const uint32_t h = iq2;
const float base = h < n_head_log2 ? m0 : m1;
const int exph = h < n_head_log2 ? h + 1 : 2*(h - n_head_log2) + 1;
slope = pow(base, exph);
}
// loop over the KV cache
// each simdgroup handles blocks of Q rows and C columns
for (int ic0 = 0; ic0 < ne11; ic0 += C*nsg) {
const int ic = ic0 + C*sgitg;
if (ic >= ne11) {
break;
}
// Q*K^T
{
for (short cc = 0; cc < C/8; ++cc) {
simdgroup_float8x8 mqk = make_filled_simdgroup_matrix(0.h);
device const half * pk = (device const half *) ((device const char *) k + ((ic + 8*cc)*nb11 + ik2*nb12 + ik3*nb13));
for (short i = 0; i < D8; ++i) {
simdgroup_half8x8 mk;
simdgroup_load(mk, pk + i*8, nb11/sizeof(half), 0, true); // transpose
simdgroup_multiply_accumulate(mqk, mq[i], mk, mqk);
}
simdgroup_store(mqk, ss + 8*cc, TF, 0, false);
}
}
// used to detect blocks full of -INF
float smax = -INFINITY;
// online softmax
{
float ms[Q];
for (short j = 0; j < Q; ++j) {
const float m = M[j];
// scale and apply the logitcap / mask
float s = ss[j*TF + tiisg]*scale;
if (logit_softcap != 0.0f) {
s = logit_softcap*precise::tanh(s);
}
if (mask != q) {
// mqk = mqk + mask*slope
s += slope*mp[ic + j*nb31/sizeof(half) + tiisg];
}
smax = simd_max(max(smax, s));
M[j] = simd_max(max(M[j], s));
ms[j] = exp(m - M[j]);
const float vs = exp(s - M[j]);
S[j] = S[j]*ms[j] + simd_sum(vs);
// the P matrix from the paper (Q rows, C columns)
ss[j*TF + tiisg] = vs;
}
// create a QxQ diagonal matrix for rescaling the output
if (tiisg < Q) {
ss[tiisg*TF + C + tiisg] = ms[tiisg];
}
}
// skip -INF blocks
if (smax == -INFINITY) {
continue;
}
// O = diag(ms)*O
{
simdgroup_float8x8 mm;
simdgroup_load(mm, ss + C, TF, 0, false);
for (short i = 0; i < D8; ++i) {
simdgroup_multiply(lo[i], mm, lo[i]);
}
}
// O = O + (Q*K^T)*V
{
for (short cc = 0; cc < C/8; ++cc) {
device const half * pv = (device const half *) ((device const char *) v + ((ic + 8*cc)*nb21 + iv2*nb22 + iv3*nb23));
for (short i = 0; i < D8; ++i) {
simdgroup_half8x8 mk;
simdgroup_load(mk, pv + i*8, nb21/sizeof(half), 0, false);
simdgroup_float8x8 mv;
simdgroup_load(mv, ss + 8*cc, TF, 0, false);
simdgroup_multiply_accumulate(lo[i], mv, mk, lo[i]);
}
}
}
}
// these are needed for reducing the results from the simdgroups (reuse the ss buffer)
for (short j = 0; j < Q; ++j) {
if (tiisg == 0) {
ss[j*TF + 0] = S[j];
ss[j*TF + 1] = M[j];
}
}
}
// reduce the warps sequentially
for (short sg = 1; sg < nsg; ++sg) {
float S = { 0.0h };
float M = { -FLT_MAX/2 };
threadgroup_barrier(mem_flags::mem_threadgroup);
// each simdgroup stores its output to shared memory, reusing sq
if (sgitg == sg) {
for (short i = 0; i < D8; ++i) {
simdgroup_store(lo[i], sq + i*8, T, 0, false);
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
// the first simdgroup accumulates the results from the other simdgroups
if (sgitg == 0) {
for (short j = 0; j < Q; ++j) {
const float S0 = ss[j*TF + 0];
const float S1 = ss[j*TF + sg*SH + 0];
const float M0 = ss[j*TF + 1];
const float M1 = ss[j*TF + sg*SH + 1];
M = max(M0, M1);
const float ms0 = exp(M0 - M);
const float ms1 = exp(M1 - M);
S = S0*ms0 + S1*ms1;
if (tiisg == 0) {
ss[j*TF + 0] = S;
ss[j*TF + 1] = M;
ss[j*TF + C + j ] = ms0;
ss[j*TF + C + j + sg*SH] = ms1;
}
}
// O_0 = diag(ms0)*O_0 + diag(ms1)*O_1
{
simdgroup_half8x8 t;
simdgroup_float8x8 ms0;
simdgroup_float8x8 ms1;
simdgroup_load(ms0, ss + C, TF, 0, false);
simdgroup_load(ms1, ss + C + sg*SH, TF, 0, false);
for (short i = 0; i < D8; ++i) {
simdgroup_load (t, sq + i*8, T, 0, false);
simdgroup_multiply(t, ms1, t);
simdgroup_multiply_accumulate(lo[i], ms0, lo[i], t);
}
}
}
}
// store result to shared memory (reuse sq)
if (sgitg == 0) {
for (short i = 0; i < D8; ++i) {
simdgroup_store(lo[i], sq + i*8, T, 0, false);
}
}
device float4 * dst4 = (device float4 *) dst;
// final rescale with 1/S and store to global memory
if (sgitg == 0) {
for (short j = 0; j < Q && iq1 + j < ne01; ++j) {
const float S = ss[j*TF + 0];
for (short i = tiisg; i < D4; i += NW) {
dst4[(iq3*ne2*ne1 + iq2 + (iq1 + j)*ne1)*D4 + i] = (float4) sq4[j*T4 + i]/S;
}
}
}
}
template [[host_name("kernel_flash_attn_ext_f16_h64" )]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<64>;
template [[host_name("kernel_flash_attn_ext_f16_h80" )]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<80>;
template [[host_name("kernel_flash_attn_ext_f16_h96" )]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<96>;
template [[host_name("kernel_flash_attn_ext_f16_h112")]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<112>;
template [[host_name("kernel_flash_attn_ext_f16_h128")]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<128>;
//template [[host_name("kernel_flash_attn_ext_f16_h256")]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_f16<256>;
template // head size, queries per threadgroup, cache items per threadgroup
kernel void kernel_flash_attn_ext_vec_f16(
device const char * q,
device const char * k,
device const char * v,
device const char * mask,
device float * dst,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant uint64_t & nb21,
constant uint64_t & nb22,
constant uint64_t & nb23,
constant uint64_t & nb31,
constant int64_t & ne1,
constant int64_t & ne2,
constant float & scale,
constant float & max_bias,
constant float & m0,
constant float & m1,
constant uint32_t & n_head_log2,
constant float & logit_softcap,
threadgroup half * shared [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]],
ushort tiisg[[thread_index_in_simdgroup]],
ushort sgitg[[simdgroup_index_in_threadgroup]]) {
const short nsg = ntg.y; // number of simdgroups
const short iq3 = tgpig[2];
const short iq2 = tgpig[1];
const short iq1 = tgpig[0];
const short D4 = D/4;
const short NW = N_SIMDWIDTH;
const short SH = (C + Q); // shared memory per simdgroup in (half)
const short T = D + 2*nsg*SH; // shared memory size per query in (half)
float slope = 1.0f;
// ALiBi
if (max_bias > 0.0f) {
const uint32_t h = iq2;
const float base = h < n_head_log2 ? m0 : m1;
const int exp = h < n_head_log2 ? h + 1 : 2*(h - n_head_log2) + 1;
slope = pow(base, exp);
}
//threadgroup half * sq = (threadgroup half *) (shared + 0*D); // holds the query data
threadgroup half4 * sq4 = (threadgroup half4 *) (shared + 0*D); // same as above but in half4
threadgroup float * ss = (threadgroup float *) (shared + 2*sgitg*SH + 1*D); // scratch buffer for attention and diagonal matrix
threadgroup float4 * ss4 = (threadgroup float4 *) (shared + 2*sgitg*SH + 1*D); // same as above but in half4
threadgroup half4 * sr4 = (threadgroup half4 *) (shared + sgitg*D + 1*T); // scratch buffer for the results
// store the result for all queries in local memory in 8x8 matrices (the O matrix from the paper)
half4 lo[D4/NW];
// load heads from Q to shared memory
device const float4 * q4 = (device const float4 *) ((device const char *) q + (iq1*nb01 + iq2*nb02 + iq3*nb03));
for (short i = tiisg; i < D4; i += NW) {
if (iq1 < ne01) {
sq4[i] = (half4) q4[i];
} else {
sq4[i] = 0.0h;
}
}
// zero out lo
for (short i = tiisg; i < D4; i += NW) {
lo[i/NW] = 0.0h;
}
// zero out shared memory SH
for (short i = tiisg; i < SH/4; i += NW) {
ss4[i] = 0.0h;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
{
float S = { 0.0h };
float M = { -FLT_MAX/2 };
// assume K and V are same shape
const short ne22 = ne12;
const short ne23 = ne13;
// broadcast
const short rk2 = ne02/ne12;
const short rk3 = ne03/ne13;
const short rv2 = ne02/ne22;
const short rv3 = ne03/ne23;
// k indices
const short ik2 = iq2 / rk2;
const short ik3 = iq3 / rk3;
// v indices
const short iv2 = iq2 / rv2;
const short iv3 = iq3 / rv3;
// load the queries from shared memory into local memory
float4 mq[D4];
for (short ii = 0; ii < D4; ii += NW) {
short i = ii + tiisg;
mq[i] = (float4) sq4[i];
}
// pointer to the mask
device const half4 * mp4 = (device const half4 *) (mask + iq1*nb31);
// loop over the KV cache
// each simdgroup handles blocks of Q rows and C columns
for (int ic0 = 0; ic0 < ne11; ic0 += C*nsg) {
const int ic = ic0 + C*sgitg;
if (ic >= ne11) {
break;
}
// Q*K^T
{
#pragma unroll
for (short cc = 0; cc < C/4; ++cc) {
float4 mqk = { 0.0h };
device const half4 * pk4 = (device const half4 *) ((device const char *) k + ((ic + 4*cc)*nb11 + ik2*nb12 + ik3*nb13));
#pragma unroll
for (short ii = 0; ii < D4; ii += NW) {
const short i = ii + tiisg;
float4x4 mk;
mk[0] = (float4) pk4[i + 0*(nb11/8)];
mk[1] = (float4) pk4[i + 1*(nb11/8)];
mk[2] = (float4) pk4[i + 2*(nb11/8)];
mk[3] = (float4) pk4[i + 3*(nb11/8)];
mqk += (float4) (mq[i] * mk);
}
// reduce the results from the threads in the simdgroup
mqk += simd_shuffle_down(mqk, 16);
mqk += simd_shuffle_down(mqk, 8);
mqk += simd_shuffle_down(mqk, 4);
mqk += simd_shuffle_down(mqk, 2);
mqk += simd_shuffle_down(mqk, 1);
// mqk = mqk*scale + mask*slope
if (tiisg == 0) {
mqk *= scale;
if (logit_softcap != 0.0f) {
mqk = logit_softcap*precise::tanh(mqk);
}
mqk += (mask != q) ? ((float4) mp4[ic/4 + cc])*slope : (float4) 0.0f;
ss4[cc] = mqk;
}
}
}
// online softmax
{
const short p = tiisg;
const float m = M;
const float s = ss[p];
M = simd_max(max(M, s));
const float ms = exp(m - M);
const float vs = exp(s - M);
S = S*ms + simd_sum(vs);
// the P matrix from the paper (Q rows, C columns)
ss[p] = vs;
// O = diag(ms)*O
#pragma unroll
for (short ii = 0; ii < D4; ii += NW) {
const short i = ii + tiisg;
lo[i/NW] *= ms;
}
}
// O = O + (Q*K^T)*V
{
#pragma unroll
for (short cc = 0; cc < C/4; ++cc) {
device const half4 * pv4 = (device const half4 *) ((device const char *) v + ((ic + 4*cc)*nb21 + iv2*nb22 + iv3*nb23));
#pragma unroll
for (short ii = 0; ii < D4; ii += NW) {
const short i = ii + tiisg;
lo[i/NW] += pv4[i + 0*(nb21/8)] * ss[4*cc + 0];
lo[i/NW] += pv4[i + 1*(nb21/8)] * ss[4*cc + 1];
lo[i/NW] += pv4[i + 2*(nb21/8)] * ss[4*cc + 2];
lo[i/NW] += pv4[i + 3*(nb21/8)] * ss[4*cc + 3];
}
}
}
}
// these are needed for reducing the results from the simdgroups (reuse the ss buffer)
if (tiisg == 0) {
ss[0] = S;
ss[1] = M;
}
}
// store results to shared memory
for (short ii = 0; ii < D4; ii += NW) {
short i = ii + tiisg;
sr4[i] = lo[ii/NW];
}
threadgroup_barrier(mem_flags::mem_threadgroup);
// parallel reduce
for (short r = nsg/2; r > 0; r >>= 1) {
if (sgitg < r) {
const float S0 = ss[ 0];
const float S1 = ss[r*SH + 0];
const float M0 = ss[ 1];
const float M1 = ss[r*SH + 1];
const float M = max(M0, M1);
const float ms0 = exp(M0 - M);
const float ms1 = exp(M1 - M);
const float S = S0*ms0 + S1*ms1;
if (tiisg == 0) {
ss[0] = S;
ss[1] = M;
}
// O_0 = diag(ms0)*O_0 + diag(ms1)*O_1
for (short ii = 0; ii < D4; ii += NW) {
short i = ii + tiisg;
sr4[i] = sr4[i]*ms0 + sr4[i + r*D4]*ms1;
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
}
device float4 * dst4 = (device float4 *) dst;
// final rescale with 1/S and store to global memory
if (sgitg == 0) {
const float S = ss[0];
for (short ii = 0; ii < D4; ii += NW) {
short i = ii + tiisg;
dst4[(iq3*ne2*ne1 + iq2 + (iq1)*ne1)*D4 + i] = (float4) sr4[i]/S;
}
}
}
template [[host_name("kernel_flash_attn_ext_vec_f16_h128")]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_vec_f16<128>;
//template [[host_name("kernel_flash_attn_ext_vec_f16_h256")]] kernel flash_attn_ext_f16_t kernel_flash_attn_ext_vec_f16<256>;
template
kernel void kernel_cpy(
device const void * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0);
device T1 * dst_data = (device T1 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x; i00 < ne00; i00 += ntg.x) {
device const T0 * src = (device T0 *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
dst_data[i00] = (T1) src[0];
}
}
typedef decltype(kernel_cpy) kernel_cpy_t;
template [[host_name("kernel_cpy_f32_f32")]] kernel kernel_cpy_t kernel_cpy;
template [[host_name("kernel_cpy_f32_f16")]] kernel kernel_cpy_t kernel_cpy;
template [[host_name("kernel_cpy_f16_f16")]] kernel kernel_cpy_t kernel_cpy;
template [[host_name("kernel_cpy_f16_f32")]] kernel kernel_cpy_t kernel_cpy;
kernel void kernel_cpy_f32_q8_0(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK8_0;
device block_q8_0 * dst_data = (device block_q8_0 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK8_0; i00 < ne00; i00 += ntg.x*QK8_0) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float amax = 0.0f; // absolute max
for (int j = 0; j < QK8_0; j++) {
const float v = src[j];
amax = MAX(amax, fabs(v));
}
const float d = amax / ((1 << 7) - 1);
const float id = d ? 1.0f/d : 0.0f;
dst_data[i00/QK8_0].d = d;
for (int j = 0; j < QK8_0; ++j) {
const float x0 = src[j]*id;
dst_data[i00/QK8_0].qs[j] = round(x0);
}
}
}
kernel void kernel_cpy_f32_q4_0(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK4_0;
device block_q4_0 * dst_data = (device block_q4_0 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK4_0; i00 < ne00; i00 += ntg.x*QK4_0) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float amax = 0.0f; // absolute max
float max = 0.0f;
for (int j = 0; j < QK4_0; j++) {
const float v = src[j];
if (amax < fabs(v)) {
amax = fabs(v);
max = v;
}
}
const float d = max / -8;
const float id = d ? 1.0f/d : 0.0f;
dst_data[i00/QK4_0].d = d;
for (int j = 0; j < QK4_0/2; ++j) {
const float x0 = src[0 + j]*id;
const float x1 = src[QK4_0/2 + j]*id;
const uint8_t xi0 = MIN(15, (int8_t)(x0 + 8.5f));
const uint8_t xi1 = MIN(15, (int8_t)(x1 + 8.5f));
dst_data[i00/QK4_0].qs[j] = xi0;
dst_data[i00/QK4_0].qs[j] |= xi1 << 4;
}
}
}
kernel void kernel_cpy_f32_q4_1(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK4_1;
device block_q4_1 * dst_data = (device block_q4_1 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK4_1; i00 < ne00; i00 += ntg.x*QK4_1) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float min = FLT_MAX;
float max = -FLT_MAX;
for (int j = 0; j < QK4_1; j++) {
const float v = src[j];
if (min > v) min = v;
if (max < v) max = v;
}
const float d = (max - min) / ((1 << 4) - 1);
const float id = d ? 1.0f/d : 0.0f;
dst_data[i00/QK4_1].d = d;
dst_data[i00/QK4_1].m = min;
for (int j = 0; j < QK4_1/2; ++j) {
const float x0 = (src[0 + j] - min)*id;
const float x1 = (src[QK4_1/2 + j] - min)*id;
const uint8_t xi0 = MIN(15, (int8_t)(x0 + 0.5f));
const uint8_t xi1 = MIN(15, (int8_t)(x1 + 0.5f));
dst_data[i00/QK4_1].qs[j] = xi0;
dst_data[i00/QK4_1].qs[j] |= xi1 << 4;
}
}
}
kernel void kernel_cpy_f32_q5_0(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK5_0;
device block_q5_0 * dst_data = (device block_q5_0 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK5_0; i00 < ne00; i00 += ntg.x*QK5_0) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float amax = 0.0f; // absolute max
float max = 0.0f;
for (int j = 0; j < QK5_0; j++) {
const float v = src[j];
if (amax < fabs(v)) {
amax = fabs(v);
max = v;
}
}
const float d = max / -16;
const float id = d ? 1.0f/d : 0.0f;
dst_data[i00/QK5_0].d = d;
uint32_t qh = 0;
for (int j = 0; j < QK5_0/2; ++j) {
const float x0 = src[0 + j]*id;
const float x1 = src[QK5_0/2 + j]*id;
const uint8_t xi0 = MIN(31, (int8_t)(x0 + 16.5f));
const uint8_t xi1 = MIN(31, (int8_t)(x1 + 16.5f));
dst_data[i00/QK5_0].qs[j] = (xi0 & 0xf) | ((xi1 & 0xf) << 4);
qh |= ((xi0 & 0x10u) >> 4) << (j + 0);
qh |= ((xi1 & 0x10u) >> 4) << (j + QK5_0/2);
}
thread const uint8_t * qh8 = (thread const uint8_t *)&qh;
for (int j = 0; j < 4; ++j) {
dst_data[i00/QK5_0].qh[j] = qh8[j];
}
}
}
kernel void kernel_cpy_f32_q5_1(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK5_1;
device block_q5_1 * dst_data = (device block_q5_1 *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK5_1; i00 < ne00; i00 += ntg.x*QK5_1) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float max = src[0];
float min = src[0];
for (int j = 1; j < QK5_1; j++) {
const float v = src[j];
min = v < min ? v : min;
max = v > max ? v : max;
}
const float d = (max - min) / 31;
const float id = d ? 1.0f/d : 0.0f;
dst_data[i00/QK5_1].d = d;
dst_data[i00/QK5_1].m = min;
uint32_t qh = 0;
for (int j = 0; j < QK5_1/2; ++j) {
const float x0 = (src[0 + j] - min)*id;
const float x1 = (src[QK5_1/2 + j] - min)*id;
const uint8_t xi0 = (uint8_t)(x0 + 0.5f);
const uint8_t xi1 = (uint8_t)(x1 + 0.5f);
dst_data[i00/QK5_1].qs[j] = (xi0 & 0xf) | ((xi1 & 0xf) << 4);
qh |= ((xi0 & 0x10u) >> 4) << (j + 0);
qh |= ((xi1 & 0x10u) >> 4) << (j + QK5_1/2);
}
thread const uint8_t * qh8 = (thread const uint8_t *)&qh;
for (int j = 0; j < 4; ++j) {
dst_data[i00/QK5_1].qh[j] = qh8[j];
}
}
}
static inline int best_index_int8(int n, constant float * val, float x) {
if (x <= val[0]) return 0;
if (x >= val[n-1]) return n-1;
int ml = 0, mu = n-1;
while (mu-ml > 1) {
int mav = (ml+mu)/2;
if (x < val[mav]) mu = mav; else ml = mav;
}
return x - val[mu-1] < val[mu] - x ? mu-1 : mu;
}
constexpr constant static float kvalues_iq4nl_f[16] = {
-127.f, -104.f, -83.f, -65.f, -49.f, -35.f, -22.f, -10.f, 1.f, 13.f, 25.f, 38.f, 53.f, 69.f, 89.f, 113.f
};
kernel void kernel_cpy_f32_iq4_nl(
device const float * src0,
device void * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i03 = tgpig[2];
const int64_t i02 = tgpig[1];
const int64_t i01 = tgpig[0];
const int64_t n = i03*ne02*ne01*ne00 + i02*ne01*ne00 + i01*ne00;
const int64_t i3 = n / (ne2*ne1*ne0);
const int64_t i2 = (n - i3*ne2*ne1*ne0) / (ne1*ne0);
const int64_t i1 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0) / ne0;
const int64_t i0 = (n - i3*ne2*ne1*ne0 - i2*ne1*ne0 - i1*ne0)/QK4_NL;
device block_iq4_nl * dst_data = (device block_iq4_nl *) ((device char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
for (int64_t i00 = tpitg.x*QK4_NL; i00 < ne00; i00 += ntg.x*QK4_NL) {
device const float * src = (device float *)((device char *) src0 + i03*nb03 + i02*nb02 + i01*nb01 + i00*nb00);
float amax = 0.0f; // absolute max
float max = 0.0f;
for (int j = 0; j < QK4_0; j++) {
const float v = src[j];
if (amax < fabs(v)) {
amax = fabs(v);
max = v;
}
}
const float d = max / kvalues_iq4nl_f[0];
const float id = d ? 1.0f/d : 0.0f;
float sumqx = 0, sumq2 = 0;
for (int j = 0; j < QK4_NL/2; ++j) {
const float x0 = src[0 + j]*id;
const float x1 = src[QK4_NL/2 + j]*id;
const uint8_t xi0 = best_index_int8(16, kvalues_iq4nl_f, x0);
const uint8_t xi1 = best_index_int8(16, kvalues_iq4nl_f, x1);
dst_data[i00/QK4_NL].qs[j] = xi0 | (xi1 << 4);
const float v0 = kvalues_iq4nl_f[xi0];
const float v1 = kvalues_iq4nl_f[xi1];
const float w0 = src[0 + j]*src[0 + j];
const float w1 = src[QK4_NL/2 + j]*src[QK4_NL/2 + j];
sumqx += w0*v0*src[j] + w1*v1*src[QK4_NL/2 + j];
sumq2 += w0*v0*v0 + w1*v1*v1;
}
dst_data[i00/QK4_NL].d = sumq2 > 0 ? sumqx/sumq2 : d;
}
}
kernel void kernel_concat(
device const char * src0,
device const char * src1,
device char * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant int64_t & ne03,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant uint64_t & nb03,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant uint64_t & nb13,
constant int64_t & ne0,
constant int64_t & ne1,
constant int64_t & ne2,
constant int64_t & ne3,
constant uint64_t & nb0,
constant uint64_t & nb1,
constant uint64_t & nb2,
constant uint64_t & nb3,
constant int32_t & dim,
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]) {
const int64_t i3 = tgpig.z;
const int64_t i2 = tgpig.y;
const int64_t i1 = tgpig.x;
int64_t o[4] = {0, 0, 0, 0};
o[dim] = dim == 0 ? ne00 : (dim == 1 ? ne01 : (dim == 2 ? ne02 : ne03));
device const float * x;
for (int i0 = tpitg.x; i0 < ne0; i0 += ntg.x) {
if (i0 < ne00 && i1 < ne01 && i2 < ne02 && i3 < ne03) {
x = (device const float *)(src0 + (i3 )*nb03 + (i2 )*nb02 + (i1 )*nb01 + (i0 )*nb00);
} else {
x = (device const float *)(src1 + (i3 - o[3])*nb13 + (i2 - o[2])*nb12 + (i1 - o[1])*nb11 + (i0 - o[0])*nb10);
}
device float * y = (device float *)(dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
*y = *x;
}
}
void kernel_mul_mv_q2_K_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q2_K * x = (device const block_q2_K *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int step = sizeof(block_q2_K) * nb;
const int ix = tiisg/8; // 0...3
const int it = tiisg%8; // 0...7
const int iq = it/4; // 0 or 1
const int ir = it%4; // 0...3
const int is = (8*ir)/16;// 0 or 1
device const float * y4 = y + ix * QK_K + 128 * iq + 8 * ir;
for (int ib = ix; ib < nb; ib += 4) {
float4 sumy = {0.f, 0.f, 0.f, 0.f};
for (int i = 0; i < 8; ++i) {
yl[i+ 0] = y4[i+ 0]; sumy[0] += yl[i+ 0];
yl[i+ 8] = y4[i+32]; sumy[1] += yl[i+ 8];
yl[i+16] = y4[i+64]; sumy[2] += yl[i+16];
yl[i+24] = y4[i+96]; sumy[3] += yl[i+24];
}
device const uint8_t * sc = (device const uint8_t *)x[ib].scales + 8*iq + is;
device const uint16_t * qs = (device const uint16_t *)x[ib].qs + 16 * iq + 4 * ir;
device const half * dh = &x[ib].d;
for (int row = 0; row < N_DST; row++) {
float4 acc1 = {0.f, 0.f, 0.f, 0.f};
float4 acc2 = {0.f, 0.f, 0.f, 0.f};
for (int i = 0; i < 8; i += 2) {
acc1[0] += yl[i+ 0] * (qs[i/2] & 0x0003);
acc2[0] += yl[i+ 1] * (qs[i/2] & 0x0300);
acc1[1] += yl[i+ 8] * (qs[i/2] & 0x000c);
acc2[1] += yl[i+ 9] * (qs[i/2] & 0x0c00);
acc1[2] += yl[i+16] * (qs[i/2] & 0x0030);
acc2[2] += yl[i+17] * (qs[i/2] & 0x3000);
acc1[3] += yl[i+24] * (qs[i/2] & 0x00c0);
acc2[3] += yl[i+25] * (qs[i/2] & 0xc000);
}
float dall = dh[0];
float dmin = dh[1] * 1.f/16.f;
sumf[row] += dall * ((acc1[0] + 1.f/256.f * acc2[0]) * (sc[0] & 0xF) * 1.f/ 1.f +
(acc1[1] + 1.f/256.f * acc2[1]) * (sc[2] & 0xF) * 1.f/ 4.f +
(acc1[2] + 1.f/256.f * acc2[2]) * (sc[4] & 0xF) * 1.f/16.f +
(acc1[3] + 1.f/256.f * acc2[3]) * (sc[6] & 0xF) * 1.f/64.f) -
dmin * (sumy[0] * (sc[0] & 0xF0) + sumy[1] * (sc[2] & 0xF0) + sumy[2] * (sc[4] & 0xF0) + sumy[3] * (sc[6] & 0xF0));
qs += step/2;
sc += step;
dh += step/2;
}
y4 += 4 * QK_K;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
[[host_name("kernel_mul_mv_q2_K_f32")]]
kernel void kernel_mul_mv_q2_K_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q2_K_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_q3_K_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int64_t r0 = tgpig.x;
const int64_t r1 = tgpig.y;
const int64_t im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * 2;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q3_K * x = (device const block_q3_K *) src0 + first_row*nb + offset0;
device const float * yy = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
//const uint16_t kmask1 = 0x3030;
//const uint16_t kmask2 = 0x0f0f;
const int tid = tiisg/4;
const int ix = tiisg%4;
const int ip = tid/4; // 0 or 1
const int il = 2*((tid%4)/2); // 0 or 2
const int ir = tid%2;
const int n = 8;
const int l0 = n*ir;
// One would think that the Metal compiler would figure out that ip and il can only have
// 4 possible states, and optimize accordingly. Well, no. It needs help, and we do it
// with these two tales.
//
// Possible masks for the high bit
const ushort4 mm[4] = {{0x0001, 0x0100, 0x0002, 0x0200}, // ip = 0, il = 0
{0x0004, 0x0400, 0x0008, 0x0800}, // ip = 0, il = 2
{0x0010, 0x1000, 0x0020, 0x2000}, // ip = 1, il = 0
{0x0040, 0x4000, 0x0080, 0x8000}}; // ip = 1, il = 2
// Possible masks for the low 2 bits
const int4 qm[2] = {{0x0003, 0x0300, 0x000c, 0x0c00}, {0x0030, 0x3000, 0x00c0, 0xc000}};
const ushort4 hm = mm[2*ip + il/2];
const int shift = 2*il;
const float v1 = il == 0 ? 4.f : 64.f;
const float v2 = 4.f * v1;
const uint16_t s_shift1 = 4*ip;
const uint16_t s_shift2 = s_shift1 + il;
const int q_offset = 32*ip + l0;
const int y_offset = 128*ip + 32*il + l0;
const int step = sizeof(block_q3_K) * nb / 2;
device const float * y1 = yy + ix*QK_K + y_offset;
uint32_t scales32, aux32;
thread uint16_t * scales16 = (thread uint16_t *)&scales32;
thread const int8_t * scales = (thread const int8_t *)&scales32;
float sumf1[2] = {0.f};
float sumf2[2] = {0.f};
for (int i = ix; i < nb; i += 4) {
for (int l = 0; l < 8; ++l) {
yl[l+ 0] = y1[l+ 0];
yl[l+ 8] = y1[l+16];
yl[l+16] = y1[l+32];
yl[l+24] = y1[l+48];
}
device const uint16_t * q = (device const uint16_t *)(x[i].qs + q_offset);
device const uint16_t * h = (device const uint16_t *)(x[i].hmask + l0);
device const uint16_t * a = (device const uint16_t *)(x[i].scales);
device const half * dh = &x[i].d;
for (int row = 0; row < 2; ++row) {
const float d_all = (float)dh[0];
scales16[0] = a[4];
scales16[1] = a[5];
aux32 = ((scales32 >> s_shift2) << 4) & 0x30303030;
scales16[0] = a[il+0];
scales16[1] = a[il+1];
scales32 = ((scales32 >> s_shift1) & 0x0f0f0f0f) | aux32;
float s1 = 0, s2 = 0, s3 = 0, s4 = 0, s5 = 0, s6 = 0;
for (int l = 0; l < n; l += 2) {
const int32_t qs = q[l/2];
s1 += yl[l+0] * (qs & qm[il/2][0]);
s2 += yl[l+1] * (qs & qm[il/2][1]);
s3 += ((h[l/2] & hm[0]) ? 0.f : yl[l+0]) + ((h[l/2] & hm[1]) ? 0.f : yl[l+1]);
s4 += yl[l+16] * (qs & qm[il/2][2]);
s5 += yl[l+17] * (qs & qm[il/2][3]);
s6 += ((h[l/2] & hm[2]) ? 0.f : yl[l+16]) + ((h[l/2] & hm[3]) ? 0.f : yl[l+17]);
}
float d1 = d_all * (s1 + 1.f/256.f * s2 - s3*v1);
float d2 = d_all * (s4 + 1.f/256.f * s5 - s6*v2);
sumf1[row] += d1 * (scales[0] - 32);
sumf2[row] += d2 * (scales[2] - 32);
s1 = s2 = s3 = s4 = s5 = s6 = 0;
for (int l = 0; l < n; l += 2) {
const int32_t qs = q[l/2+8];
s1 += yl[l+8] * (qs & qm[il/2][0]);
s2 += yl[l+9] * (qs & qm[il/2][1]);
s3 += ((h[l/2+8] & hm[0]) ? 0.f : yl[l+8]) + ((h[l/2+8] & hm[1]) ? 0.f : yl[l+9]);
s4 += yl[l+24] * (qs & qm[il/2][2]);
s5 += yl[l+25] * (qs & qm[il/2][3]);
s6 += ((h[l/2+8] & hm[2]) ? 0.f : yl[l+24]) + ((h[l/2+8] & hm[3]) ? 0.f : yl[l+25]);
}
d1 = d_all * (s1 + 1.f/256.f * s2 - s3*v1);
d2 = d_all * (s4 + 1.f/256.f * s5 - s6*v2);
sumf1[row] += d1 * (scales[1] - 32);
sumf2[row] += d2 * (scales[3] - 32);
q += step;
h += step;
a += step;
dh += step;
}
y1 += 4 * QK_K;
}
for (int row = 0; row < 2; ++row) {
const float sumf = (sumf1[row] + 0.25f * sumf2[row]) / (1 << shift);
sumf1[row] = simd_sum(sumf);
}
if (tiisg == 0) {
for (int row = 0; row < 2; ++row) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = sumf1[row];
}
}
}
[[host_name("kernel_mul_mv_q3_K_f32")]]
kernel void kernel_mul_mv_q3_K_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q3_K_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_q4_K_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const uint16_t kmask1 = 0x3f3f;
const uint16_t kmask2 = 0x0f0f;
const uint16_t kmask3 = 0xc0c0;
const int ix = tiisg/8; // 0...3
const int it = tiisg%8; // 0...7
const int iq = it/4; // 0 or 1
const int ir = it%4; // 0...3
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
//const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int first_row = r0 * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q4_K * x = (device const block_q4_K *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[16];
float yh[16];
float sumf[N_DST]={0.f}, all_sum;
const int step = sizeof(block_q4_K) * nb / 2;
device const float * y4 = y + ix * QK_K + 64 * iq + 8 * ir;
uint16_t sc16[4];
thread const uint8_t * sc8 = (thread const uint8_t *)sc16;
for (int ib = ix; ib < nb; ib += 4) {
float4 sumy = {0.f, 0.f, 0.f, 0.f};
for (int i = 0; i < 8; ++i) {
yl[i+0] = y4[i+ 0]; sumy[0] += yl[i+0];
yl[i+8] = y4[i+ 32]; sumy[1] += yl[i+8];
yh[i+0] = y4[i+128]; sumy[2] += yh[i+0];
yh[i+8] = y4[i+160]; sumy[3] += yh[i+8];
}
device const uint16_t * sc = (device const uint16_t *)x[ib].scales + iq;
device const uint16_t * q1 = (device const uint16_t *)x[ib].qs + 16 * iq + 4 * ir;
device const half * dh = &x[ib].d;
for (int row = 0; row < N_DST; row++) {
sc16[0] = sc[0] & kmask1;
sc16[1] = sc[2] & kmask1;
sc16[2] = ((sc[4] >> 0) & kmask2) | ((sc[0] & kmask3) >> 2);
sc16[3] = ((sc[4] >> 4) & kmask2) | ((sc[2] & kmask3) >> 2);
device const uint16_t * q2 = q1 + 32;
float4 acc1 = {0.f, 0.f, 0.f, 0.f};
float4 acc2 = {0.f, 0.f, 0.f, 0.f};
for (int i = 0; i < 8; i += 2) {
acc1[0] += yl[i+0] * (q1[i/2] & 0x000F);
acc1[1] += yl[i+1] * (q1[i/2] & 0x0F00);
acc1[2] += yl[i+8] * (q1[i/2] & 0x00F0);
acc1[3] += yl[i+9] * (q1[i/2] & 0xF000);
acc2[0] += yh[i+0] * (q2[i/2] & 0x000F);
acc2[1] += yh[i+1] * (q2[i/2] & 0x0F00);
acc2[2] += yh[i+8] * (q2[i/2] & 0x00F0);
acc2[3] += yh[i+9] * (q2[i/2] & 0xF000);
}
float dall = dh[0];
float dmin = dh[1];
sumf[row] += dall * ((acc1[0] + 1.f/256.f * acc1[1]) * sc8[0] +
(acc1[2] + 1.f/256.f * acc1[3]) * sc8[1] * 1.f/16.f +
(acc2[0] + 1.f/256.f * acc2[1]) * sc8[4] +
(acc2[2] + 1.f/256.f * acc2[3]) * sc8[5] * 1.f/16.f) -
dmin * (sumy[0] * sc8[2] + sumy[1] * sc8[3] + sumy[2] * sc8[6] + sumy[3] * sc8[7]);
q1 += step;
sc += step;
dh += step;
}
y4 += 4 * QK_K;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
[[host_name("kernel_mul_mv_q4_K_f32")]]
kernel void kernel_mul_mv_q4_K_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q4_K_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_q5_K_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int64_t r0 = tgpig.x;
const int64_t r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * 2;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q5_K * x = (device const block_q5_K *) src0 + first_row*nb + offset0;
device const float * yy = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float sumf[2]={0.f};
const int step = sizeof(block_q5_K) * nb;
float yl[16], yh[16];
const uint16_t kmask1 = 0x3f3f;
const uint16_t kmask2 = 0x0f0f;
const uint16_t kmask3 = 0xc0c0;
const int tid = tiisg/4;
const int ix = tiisg%4;
const int iq = tid/4;
const int ir = tid%4;
const int n = 8;
const int l0 = n*ir;
const int q_offset = 32*iq + l0;
const int y_offset = 64*iq + l0;
const uint8_t hm1 = 1u << (2*iq);
const uint8_t hm2 = hm1 << 1;
const uint8_t hm3 = hm1 << 4;
const uint8_t hm4 = hm2 << 4;
uint16_t sc16[4];
thread const uint8_t * sc8 = (thread const uint8_t *)sc16;
device const float * y1 = yy + ix*QK_K + y_offset;
for (int i = ix; i < nb; i += 4) {
device const uint8_t * q1 = x[i].qs + q_offset;
device const uint8_t * qh = x[i].qh + l0;
device const half * dh = &x[i].d;
device const uint16_t * a = (device const uint16_t *)x[i].scales + iq;
device const float * y2 = y1 + 128;
float4 sumy = {0.f, 0.f, 0.f, 0.f};
for (int l = 0; l < 8; ++l) {
yl[l+0] = y1[l+ 0]; sumy[0] += yl[l+0];
yl[l+8] = y1[l+32]; sumy[1] += yl[l+8];
yh[l+0] = y2[l+ 0]; sumy[2] += yh[l+0];
yh[l+8] = y2[l+32]; sumy[3] += yh[l+8];
}
for (int row = 0; row < 2; ++row) {
device const uint8_t * q2 = q1 + 64;
sc16[0] = a[0] & kmask1;
sc16[1] = a[2] & kmask1;
sc16[2] = ((a[4] >> 0) & kmask2) | ((a[0] & kmask3) >> 2);
sc16[3] = ((a[4] >> 4) & kmask2) | ((a[2] & kmask3) >> 2);
float4 acc1 = {0.f};
float4 acc2 = {0.f};
for (int l = 0; l < n; ++l) {
uint8_t h = qh[l];
acc1[0] += yl[l+0] * (q1[l] & 0x0F);
acc1[1] += yl[l+8] * (q1[l] & 0xF0);
acc1[2] += yh[l+0] * (q2[l] & 0x0F);
acc1[3] += yh[l+8] * (q2[l] & 0xF0);
acc2[0] += h & hm1 ? yl[l+0] : 0.f;
acc2[1] += h & hm2 ? yl[l+8] : 0.f;
acc2[2] += h & hm3 ? yh[l+0] : 0.f;
acc2[3] += h & hm4 ? yh[l+8] : 0.f;
}
const float dall = dh[0];
const float dmin = dh[1];
sumf[row] += dall * (sc8[0] * (acc1[0] + 16.f*acc2[0]) +
sc8[1] * (acc1[1]/16.f + 16.f*acc2[1]) +
sc8[4] * (acc1[2] + 16.f*acc2[2]) +
sc8[5] * (acc1[3]/16.f + 16.f*acc2[3])) -
dmin * (sumy[0] * sc8[2] + sumy[1] * sc8[3] + sumy[2] * sc8[6] + sumy[3] * sc8[7]);
q1 += step;
qh += step;
dh += step/2;
a += step/2;
}
y1 += 4 * QK_K;
}
for (int row = 0; row < 2; ++row) {
const float tot = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = tot;
}
}
}
[[host_name("kernel_mul_mv_q5_K_f32")]]
kernel void kernel_mul_mv_q5_K_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q5_K_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_q6_K_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const uint8_t kmask1 = 0x03;
const uint8_t kmask2 = 0x0C;
const uint8_t kmask3 = 0x30;
const uint8_t kmask4 = 0xC0;
const int nb = ne00/QK_K;
const int64_t r0 = tgpig.x;
const int64_t r1 = tgpig.y;
const int im = tgpig.z;
const int row = 2 * r0 + sgitg;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_q6_K * x = (device const block_q6_K *) src0 + row * nb + offset0;
device const float * yy = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float sumf = 0;
const int tid = tiisg/2;
const int ix = tiisg%2;
const int ip = tid/8; // 0 or 1
const int il = tid%8;
const int n = 4;
const int l0 = n*il;
const int is = 8*ip + l0/16;
const int y_offset = 128*ip + l0;
const int q_offset_l = 64*ip + l0;
const int q_offset_h = 32*ip + l0;
for (int i = ix; i < nb; i += 2) {
device const uint8_t * q1 = x[i].ql + q_offset_l;
device const uint8_t * q2 = q1 + 32;
device const uint8_t * qh = x[i].qh + q_offset_h;
device const int8_t * sc = x[i].scales + is;
device const float * y = yy + i * QK_K + y_offset;
const float dall = x[i].d;
float4 sums = {0.f, 0.f, 0.f, 0.f};
for (int l = 0; l < n; ++l) {
sums[0] += y[l+ 0] * ((int8_t)((q1[l] & 0xF) | ((qh[l] & kmask1) << 4)) - 32);
sums[1] += y[l+32] * ((int8_t)((q2[l] & 0xF) | ((qh[l] & kmask2) << 2)) - 32);
sums[2] += y[l+64] * ((int8_t)((q1[l] >> 4) | ((qh[l] & kmask3) << 0)) - 32);
sums[3] += y[l+96] * ((int8_t)((q2[l] >> 4) | ((qh[l] & kmask4) >> 2)) - 32);
}
sumf += dall * (sums[0] * sc[0] + sums[1] * sc[2] + sums[2] * sc[4] + sums[3] * sc[6]);
}
const float tot = simd_sum(sumf);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + row] = tot;
}
}
[[host_name("kernel_mul_mv_q6_K_f32")]]
kernel void kernel_mul_mv_q6_K_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_q6_K_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
// ======================= "True" 2-bit
void kernel_mul_mv_iq2_xxs_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq2_xxs * x = (device const block_iq2_xxs *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
threadgroup uint64_t * values = (threadgroup uint64_t *)shared_values;
threadgroup uint8_t * shared_signs = (threadgroup uint8_t *)(values + 256);
{
int nval = 4;
int pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) values[pos + i] = iq2xxs_grid[pos + i];
nval = 2;
pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) shared_signs[pos+i] = ksigns_iq2xs[pos+i];
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq2_xxs * xr = x + ibl;
device const uint16_t * q2 = xr->qs + 4 * ib;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
const float db = dh[0];
device const uint8_t * aux8 = (device const uint8_t *)q2;
const uint32_t aux32 = q2[2] | (q2[3] << 16);
const float d = db * (0.5f + (aux32 >> 28));
float sum = 0;
for (int l = 0; l < 4; ++l) {
const threadgroup uint8_t * grid = (const threadgroup uint8_t *)(values + aux8[l]);
const uint8_t signs = shared_signs[(aux32 >> 7*l) & 127];
for (int j = 0; j < 8; ++j) {
sum += yl[8*l + j] * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f);
}
}
sumf[row] += d * sum;
dh += nb*sizeof(block_iq2_xxs)/2;
q2 += nb*sizeof(block_iq2_xxs)/2;
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum * 0.25f;
}
}
}
[[host_name("kernel_mul_mv_iq2_xxs_f32")]]
kernel void kernel_mul_mv_iq2_xxs_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq2_xxs_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_iq2_xs_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq2_xs * x = (device const block_iq2_xs *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
threadgroup uint64_t * values = (threadgroup uint64_t *)shared_values;
threadgroup uint8_t * shared_signs = (threadgroup uint8_t *)(values + 512);
{
int nval = 8;
int pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) values[pos + i] = iq2xs_grid[pos + i];
nval = 2;
pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) shared_signs[pos+i] = ksigns_iq2xs[pos+i];
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq2_xs * xr = x + ibl;
device const uint16_t * q2 = xr->qs + 4 * ib;
device const uint8_t * sc = xr->scales + ib;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
const float db = dh[0];
const uint8_t ls1 = sc[0] & 0xf;
const uint8_t ls2 = sc[0] >> 4;
const float d1 = db * (0.5f + ls1);
const float d2 = db * (0.5f + ls2);
float sum1 = 0, sum2 = 0;
for (int l = 0; l < 2; ++l) {
const threadgroup uint8_t * grid = (const threadgroup uint8_t *)(values + (q2[l] & 511));
const uint8_t signs = shared_signs[(q2[l] >> 9)];
for (int j = 0; j < 8; ++j) {
sum1 += yl[8*l + j] * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f);
}
}
for (int l = 2; l < 4; ++l) {
const threadgroup uint8_t * grid = (const threadgroup uint8_t *)(values + (q2[l] & 511));
const uint8_t signs = shared_signs[(q2[l] >> 9)];
for (int j = 0; j < 8; ++j) {
sum2 += yl[8*l + j] * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f);
}
}
sumf[row] += d1 * sum1 + d2 * sum2;
dh += nb*sizeof(block_iq2_xs)/2;
q2 += nb*sizeof(block_iq2_xs)/2;
sc += nb*sizeof(block_iq2_xs);
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum * 0.25f;
}
}
}
[[host_name("kernel_mul_mv_iq2_xs_f32")]]
kernel void kernel_mul_mv_iq2_xs_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq2_xs_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_iq3_xxs_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq3_xxs * x = (device const block_iq3_xxs *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
threadgroup uint32_t * values = (threadgroup uint32_t *)shared_values;
threadgroup uint8_t * shared_signs = (threadgroup uint8_t *)(values + 256);
{
int nval = 4;
int pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) values[pos + i] = iq3xxs_grid[pos + i];
nval = 2;
pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) shared_signs[pos+i] = ksigns_iq2xs[pos+i];
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq3_xxs * xr = x + ibl;
device const uint8_t * q3 = xr->qs + 8 * ib;
device const uint16_t * gas = (device const uint16_t *)(xr->qs + QK_K/4) + 2 * ib;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
const float db = dh[0];
const uint32_t aux32 = gas[0] | (gas[1] << 16);
const float d = db * (0.5f + (aux32 >> 28));
float2 sum = {0};
for (int l = 0; l < 4; ++l) {
const threadgroup uint8_t * grid1 = (const threadgroup uint8_t *)(values + q3[2*l+0]);
const threadgroup uint8_t * grid2 = (const threadgroup uint8_t *)(values + q3[2*l+1]);
const uint8_t signs = shared_signs[(aux32 >> 7*l) & 127];
for (int j = 0; j < 4; ++j) {
sum[0] += yl[8*l + j + 0] * grid1[j] * (signs & kmask_iq2xs[j+0] ? -1.f : 1.f);
sum[1] += yl[8*l + j + 4] * grid2[j] * (signs & kmask_iq2xs[j+4] ? -1.f : 1.f);
}
}
sumf[row] += d * (sum[0] + sum[1]);
dh += nb*sizeof(block_iq3_xxs)/2;
q3 += nb*sizeof(block_iq3_xxs);
gas += nb*sizeof(block_iq3_xxs)/2;
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum * 0.5f;
}
}
}
[[host_name("kernel_mul_mv_iq3_xxs_f32")]]
kernel void kernel_mul_mv_iq3_xxs_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq3_xxs_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_iq3_s_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq3_s * x = (device const block_iq3_s *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
threadgroup uint32_t * values = (threadgroup uint32_t *)shared_values;
{
int nval = 8;
int pos = (32*sgitg + tiisg)*nval;
for (int i = 0; i < nval; ++i) values[pos + i] = iq3s_grid[pos + i];
threadgroup_barrier(mem_flags::mem_threadgroup);
}
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq3_s * xr = x + ibl;
device const uint8_t * qs = xr->qs + 8 * ib;
device const uint8_t * qh = xr->qh + ib;
device const uint8_t * sc = xr->scales + (ib/2);
device const uint8_t * signs = xr->signs + 4 * ib;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
const float db = dh[0];
const float d = db * (1 + 2*((sc[0] >> 4*(ib%2)) & 0xf));
float2 sum = {0};
for (int l = 0; l < 4; ++l) {
const threadgroup uint32_t * table1 = qh[0] & kmask_iq2xs[2*l+0] ? values + 256 : values;
const threadgroup uint32_t * table2 = qh[0] & kmask_iq2xs[2*l+1] ? values + 256 : values;
const threadgroup uint8_t * grid1 = (const threadgroup uint8_t *)(table1 + qs[2*l+0]);
const threadgroup uint8_t * grid2 = (const threadgroup uint8_t *)(table2 + qs[2*l+1]);
for (int j = 0; j < 4; ++j) {
sum[0] += yl[8*l + j + 0] * grid1[j] * select(1, -1, signs[l] & kmask_iq2xs[j+0]);
sum[1] += yl[8*l + j + 4] * grid2[j] * select(1, -1, signs[l] & kmask_iq2xs[j+4]);
}
}
sumf[row] += d * (sum[0] + sum[1]);
dh += nb*sizeof(block_iq3_s)/2;
qs += nb*sizeof(block_iq3_s);
qh += nb*sizeof(block_iq3_s);
sc += nb*sizeof(block_iq3_s);
signs += nb*sizeof(block_iq3_s);
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
[[host_name("kernel_mul_mv_iq3_s_f32")]]
kernel void kernel_mul_mv_iq3_s_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq3_s_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_iq2_s_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq2_s * x = (device const block_iq2_s *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
//threadgroup uint64_t * values = (threadgroup uint64_t *)shared_values;
//{
// int nval = 32;
// int pos = (32*sgitg + tiisg)*nval;
// for (int i = 0; i < nval; ++i) values[pos + i] = iq2s_grid[pos + i];
// threadgroup_barrier(mem_flags::mem_threadgroup);
//}
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq2_s * xr = x + ibl;
device const uint8_t * qs = xr->qs + 4 * ib;
device const uint8_t * qh = xr->qh + ib;
device const uint8_t * sc = xr->scales + ib;
device const uint8_t * signs = qs + QK_K/8;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
const float db = dh[0];
const float d1 = db * (0.5f + (sc[0] & 0xf));
const float d2 = db * (0.5f + (sc[0] >> 4));
float2 sum = {0};
for (int l = 0; l < 2; ++l) {
//const threadgroup uint8_t * grid1 = (const threadgroup uint8_t *)(values + (qs[l+0] | ((qh[0] << (8-2*l)) & 0x300)));
//const threadgroup uint8_t * grid2 = (const threadgroup uint8_t *)(values + (qs[l+2] | ((qh[0] << (4-2*l)) & 0x300)));
constant uint8_t * grid1 = (constant uint8_t *)(iq2s_grid + (qs[l+0] | ((qh[0] << (8-2*l)) & 0x300)));
constant uint8_t * grid2 = (constant uint8_t *)(iq2s_grid + (qs[l+2] | ((qh[0] << (4-2*l)) & 0x300)));
for (int j = 0; j < 8; ++j) {
sum[0] += yl[8*l + j + 0] * grid1[j] * select(1, -1, signs[l+0] & kmask_iq2xs[j]);
sum[1] += yl[8*l + j + 16] * grid2[j] * select(1, -1, signs[l+2] & kmask_iq2xs[j]);
}
}
sumf[row] += d1 * sum[0] + d2 * sum[1];
dh += nb*sizeof(block_iq2_s)/2;
qs += nb*sizeof(block_iq2_s);
qh += nb*sizeof(block_iq2_s);
sc += nb*sizeof(block_iq2_s);
signs += nb*sizeof(block_iq2_s);
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum * 0.25f;
}
}
}
[[host_name("kernel_mul_mv_iq2_s_f32")]]
kernel void kernel_mul_mv_iq2_s_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq2_s_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
void kernel_mul_mv_iq1_s_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_value,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq1_s * x = (device const block_iq1_s *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
float sumy = 0;
for (int i = 0; i < 32; ++i) {
yl[i] = y4[i];
sumy += yl[i];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq1_s * xr = x + ibl;
device const uint8_t * qs = xr->qs + 4 * ib;
device const uint16_t * qh = xr->qh + ib;
device const half * dh = &xr->d;
for (int row = 0; row < N_DST; row++) {
constant uint8_t * grid1 = (constant uint8_t *)(iq1s_grid_gpu + (qs[0] | ((qh[0] << 8) & 0x700)));
constant uint8_t * grid2 = (constant uint8_t *)(iq1s_grid_gpu + (qs[1] | ((qh[0] << 5) & 0x700)));
constant uint8_t * grid3 = (constant uint8_t *)(iq1s_grid_gpu + (qs[2] | ((qh[0] << 2) & 0x700)));
constant uint8_t * grid4 = (constant uint8_t *)(iq1s_grid_gpu + (qs[3] | ((qh[0] >> 1) & 0x700)));
float sum = 0;
for (int j = 0; j < 4; ++j) {
sum += yl[j+ 0] * (grid1[j] & 0xf) + yl[j+ 4] * (grid1[j] >> 4)
+ yl[j+ 8] * (grid2[j] & 0xf) + yl[j+12] * (grid2[j] >> 4)
+ yl[j+16] * (grid3[j] & 0xf) + yl[j+20] * (grid3[j] >> 4)
+ yl[j+24] * (grid4[j] & 0xf) + yl[j+28] * (grid4[j] >> 4);
}
sumf[row] += (float)dh[0] * (sum + sumy * (qh[0] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA)) * (2*((qh[0] >> 12) & 7) + 1);
dh += nb*sizeof(block_iq1_s)/2;
qs += nb*sizeof(block_iq1_s);
qh += nb*sizeof(block_iq1_s)/2;
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
void kernel_mul_mv_iq1_m_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_value,
uint3 tgpig,
uint tiisg,
uint sgitg) {
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * N_SIMDGROUP + sgitg) * N_DST;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq1_m * x = (device const block_iq1_m *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
float yl[32];
float sumf[N_DST]={0.f}, all_sum;
const int nb32 = nb * (QK_K / 32);
const int ix = tiisg;
device const float * y4 = y + 32 * ix;
iq1m_scale_t scale;
for (int ib32 = ix; ib32 < nb32; ib32 += 32) {
float4 sumy = {0.f};
for (int i = 0; i < 8; ++i) {
yl[i+ 0] = y4[i+ 0]; sumy[0] += yl[i+ 0];
yl[i+ 8] = y4[i+ 8]; sumy[1] += yl[i+ 8];
yl[i+16] = y4[i+16]; sumy[2] += yl[i+16];
yl[i+24] = y4[i+24]; sumy[3] += yl[i+24];
}
const int ibl = ib32 / (QK_K / 32);
const int ib = ib32 % (QK_K / 32);
device const block_iq1_m * xr = x + ibl;
device const uint8_t * qs = xr->qs + 4 * ib;
device const uint8_t * qh = xr->qh + 2 * ib;
device const uint16_t * sc = (device const uint16_t *)xr->scales;
for (int row = 0; row < N_DST; row++) {
scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00f0) | ((sc[2] >> 4) & 0x0f00) | (sc[3] & 0xf000);
constant uint8_t * grid1 = (constant uint8_t *)(iq1s_grid_gpu + (qs[0] | ((qh[0] << 8) & 0x700)));
constant uint8_t * grid2 = (constant uint8_t *)(iq1s_grid_gpu + (qs[1] | ((qh[0] << 4) & 0x700)));
constant uint8_t * grid3 = (constant uint8_t *)(iq1s_grid_gpu + (qs[2] | ((qh[1] << 8) & 0x700)));
constant uint8_t * grid4 = (constant uint8_t *)(iq1s_grid_gpu + (qs[3] | ((qh[1] << 4) & 0x700)));
float2 sum = {0.f};
for (int j = 0; j < 4; ++j) {
sum[0] += yl[j+ 0] * (grid1[j] & 0xf) + yl[j+ 4] * (grid1[j] >> 4)
+ yl[j+ 8] * (grid2[j] & 0xf) + yl[j+12] * (grid2[j] >> 4);
sum[1] += yl[j+16] * (grid3[j] & 0xf) + yl[j+20] * (grid3[j] >> 4)
+ yl[j+24] * (grid4[j] & 0xf) + yl[j+28] * (grid4[j] >> 4);
}
const float delta1 = sumy[0] * (qh[0] & 0x08 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA) + sumy[1] * (qh[0] & 0x80 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA);
const float delta2 = sumy[2] * (qh[1] & 0x08 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA) + sumy[3] * (qh[1] & 0x80 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA);
sumf[row] += (float)scale.f16 * ((sum[0] + delta1) * (2*((sc[ib/2] >> (6*(ib%2)+0)) & 7) + 1) +
(sum[1] + delta2) * (2*((sc[ib/2] >> (6*(ib%2)+3)) & 7) + 1));
sc += nb*sizeof(block_iq1_m)/2;
qs += nb*sizeof(block_iq1_m);
qh += nb*sizeof(block_iq1_m);
}
y4 += 32 * 32;
}
for (int row = 0; row < N_DST; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
void kernel_mul_mv_iq4_nl_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values_i8,
uint3 tgpig,
uint tiisg,
uint sgitg) {
threadgroup float * shared_values = (threadgroup float *)shared_values_i8;
const int nb = ne00/QK4_NL;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * 2 + sgitg) * 2;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq4_nl * x = (device const block_iq4_nl *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
const int ix = tiisg/2; // 0...15
const int it = tiisg%2; // 0 or 1
shared_values[tiisg] = kvalues_iq4nl_f[tiisg%16];
threadgroup_barrier(mem_flags::mem_threadgroup);
float4 yl[4];
float sumf[2]={0.f}, all_sum;
device const float * yb = y + ix * QK4_NL + it * 8;
uint32_t aux32[2];
thread const uint8_t * q8 = (thread const uint8_t *)aux32;
float4 qf1, qf2;
for (int ib = ix; ib < nb; ib += 16) {
device const float4 * y4 = (device const float4 *)yb;
yl[0] = y4[0]; yl[1] = y4[4]; yl[2] = y4[1]; yl[3] = y4[5];
for (int row = 0; row < 2 && first_row + row < ne01; ++row) {
device const block_iq4_nl & xb = x[row*nb + ib];
device const uint16_t * q4 = (device const uint16_t *)(xb.qs + 8*it);
float4 acc1 = {0.f}, acc2 = {0.f};
aux32[0] = q4[0] | (q4[1] << 16);
aux32[1] = (aux32[0] >> 4) & 0x0f0f0f0f;
aux32[0] &= 0x0f0f0f0f;
qf1 = {shared_values[q8[0]], shared_values[q8[1]], shared_values[q8[2]], shared_values[q8[3]]};
qf2 = {shared_values[q8[4]], shared_values[q8[5]], shared_values[q8[6]], shared_values[q8[7]]};
acc1 += yl[0] * qf1;
acc2 += yl[1] * qf2;
aux32[0] = q4[2] | (q4[3] << 16);
aux32[1] = (aux32[0] >> 4) & 0x0f0f0f0f;
aux32[0] &= 0x0f0f0f0f;
qf1 = {shared_values[q8[0]], shared_values[q8[1]], shared_values[q8[2]], shared_values[q8[3]]};
qf2 = {shared_values[q8[4]], shared_values[q8[5]], shared_values[q8[6]], shared_values[q8[7]]};
acc1 += yl[2] * qf1;
acc2 += yl[3] * qf2;
acc1 += acc2;
sumf[row] += (float)xb.d * (acc1[0] + acc1[1] + acc1[2] + acc1[3]);
}
yb += 16 * QK4_NL;
}
for (int row = 0; row < 2 && first_row + row < ne01; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
void kernel_mul_mv_iq4_xs_f32_impl(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values_i8,
uint3 tgpig,
uint tiisg,
uint sgitg) {
threadgroup float * shared_values = (threadgroup float *)shared_values_i8;
const int nb = ne00/QK_K;
const int r0 = tgpig.x;
const int r1 = tgpig.y;
const int im = tgpig.z;
const int first_row = (r0 * 2 + sgitg) * 2;
const int ib_row = first_row * nb;
const uint i12 = im%ne12;
const uint i13 = im/ne12;
const uint offset0 = (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
device const block_iq4_xs * x = (device const block_iq4_xs *) src0 + ib_row + offset0;
device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;
const int ix = tiisg/16; // 0 or 1
const int it = tiisg%16; // 0...15
const int ib = it/2;
const int il = it%2;
shared_values[tiisg] = kvalues_iq4nl_f[tiisg%16];
threadgroup_barrier(mem_flags::mem_threadgroup);
float4 yl[4];
float sumf[2]={0.f}, all_sum;
device const float * yb = y + ix * QK_K + ib * 32 + il * 8;
uint32_t aux32[2];
thread const uint8_t * q8 = (thread const uint8_t *)aux32;
float4 qf1, qf2;
for (int ibl = ix; ibl < nb; ibl += 2) {
device const float4 * y4 = (device const float4 *)yb;
yl[0] = y4[0]; yl[1] = y4[4]; yl[2] = y4[1]; yl[3] = y4[5];
for (int row = 0; row < 2; ++row) {
device const block_iq4_xs & xb = x[row*nb + ibl];
device const uint32_t * q4 = (device const uint32_t *)(xb.qs + 16*ib + 8*il);
float4 acc1 = {0.f}, acc2 = {0.f};
aux32[0] = q4[0] & 0x0f0f0f0f;
aux32[1] = (q4[0] >> 4) & 0x0f0f0f0f;
qf1 = {shared_values[q8[0]], shared_values[q8[1]], shared_values[q8[2]], shared_values[q8[3]]};
qf2 = {shared_values[q8[4]], shared_values[q8[5]], shared_values[q8[6]], shared_values[q8[7]]};
acc1 += yl[0] * qf1;
acc2 += yl[1] * qf2;
aux32[0] = q4[1] & 0x0f0f0f0f;
aux32[1] = (q4[1] >> 4) & 0x0f0f0f0f;
qf1 = {shared_values[q8[0]], shared_values[q8[1]], shared_values[q8[2]], shared_values[q8[3]]};
qf2 = {shared_values[q8[4]], shared_values[q8[5]], shared_values[q8[6]], shared_values[q8[7]]};
acc1 += yl[2] * qf1;
acc2 += yl[3] * qf2;
acc1 += acc2;
const int ls = (((xb.scales_l[ib/2] >> 4*(ib%2)) & 0xf) | (((xb.scales_h >> 2*ib) & 3) << 4)) - 32;
sumf[row] += (float)xb.d * ls * (acc1[0] + acc1[1] + acc1[2] + acc1[3]);
}
yb += 2 * QK_K;
}
for (int row = 0; row < 2; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
}
}
}
[[host_name("kernel_mul_mv_iq1_s_f32")]]
kernel void kernel_mul_mv_iq1_s_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq1_s_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
[[host_name("kernel_mul_mv_iq1_m_f32")]]
kernel void kernel_mul_mv_iq1_m_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq1_m_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, nullptr, tgpig, tiisg, sgitg);
}
[[host_name("kernel_mul_mv_iq4_nl_f32")]]
kernel void kernel_mul_mv_iq4_nl_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq4_nl_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
[[host_name("kernel_mul_mv_iq4_xs_f32")]]
kernel void kernel_mul_mv_iq4_xs_f32(
device const void * src0,
device const float * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
kernel_mul_mv_iq4_xs_f32_impl(src0, src1, dst, ne00, ne01, ne02, ne10, ne12, ne0, ne1, r2, r3, shared_values, tgpig, tiisg, sgitg);
}
//============================= templates and their specializations =============================
// NOTE: this is not dequantizing - we are simply fitting the template
template
void dequantize_f32(device const float4x4 * src, short il, thread type4x4 & reg) {
float4x4 temp = *(((device float4x4 *)src));
for (int i = 0; i < 16; i++){
reg[i/4][i%4] = temp[i/4][i%4];
}
}
template
void dequantize_f16(device const half4x4 * src, short il, thread type4x4 & reg) {
half4x4 temp = *(((device half4x4 *)src));
for (int i = 0; i < 16; i++){
reg[i/4][i%4] = temp[i/4][i%4];
}
}
template
void dequantize_q4_0(device const block_q4_0 *xb, short il, thread type4x4 & reg) {
device const uint16_t * qs = ((device const uint16_t *)xb + 1);
const float d1 = il ? (xb->d / 16.h) : xb->d;
const float d2 = d1 / 256.f;
const float md = -8.h * xb->d;
const ushort mask0 = il ? 0x00F0 : 0x000F;
const ushort mask1 = mask0 << 8;
for (int i=0;i<8;i++) {
reg[i/2][2*(i%2)+0] = d1 * (qs[i] & mask0) + md;
reg[i/2][2*(i%2)+1] = d2 * (qs[i] & mask1) + md;
}
}
template
void dequantize_q4_1(device const block_q4_1 *xb, short il, thread type4x4 & reg) {
device const uint16_t * qs = ((device const uint16_t *)xb + 2);
const float d1 = il ? (xb->d / 16.h) : xb->d;
const float d2 = d1 / 256.f;
const float m = xb->m;
const ushort mask0 = il ? 0x00F0 : 0x000F;
const ushort mask1 = mask0 << 8;
for (int i=0;i<8;i++) {
reg[i/2][2*(i%2)+0] = ((qs[i] & mask0) * d1) + m;
reg[i/2][2*(i%2)+1] = ((qs[i] & mask1) * d2) + m;
}
}
template
void dequantize_q5_0(device const block_q5_0 *xb, short il, thread type4x4 & reg) {
device const uint16_t * qs = ((device const uint16_t *)xb + 3);
const float d = xb->d;
const float md = -16.h * xb->d;
const ushort mask = il ? 0x00F0 : 0x000F;
const uint32_t qh = *((device const uint32_t *)xb->qh);
const int x_mv = il ? 4 : 0;
const int gh_mv = il ? 12 : 0;
const int gh_bk = il ? 0 : 4;
for (int i = 0; i < 8; i++) {
// extract the 5-th bits for x0 and x1
const uint8_t xh_0 = ((qh >> (gh_mv + 2*i )) << gh_bk) & 0x10;
const uint8_t xh_1 = ((qh >> (gh_mv + 2*i+1)) << gh_bk) & 0x10;
// combine the 4-bits from qs with the 5th bit
const int32_t x0 = ((((qs[i] ) & mask) >> x_mv) | xh_0);
const int32_t x1 = ((((qs[i] >> 8) & mask) >> x_mv) | xh_1);
reg[i/2][2*(i%2)+0] = d * x0 + md;
reg[i/2][2*(i%2)+1] = d * x1 + md;
}
}
template
void dequantize_q5_1(device const block_q5_1 *xb, short il, thread type4x4 & reg) {
device const uint16_t * qs = ((device const uint16_t *)xb + 4);
const float d = xb->d;
const float m = xb->m;
const ushort mask = il ? 0x00F0 : 0x000F;
const uint32_t qh = *((device const uint32_t *)xb->qh);
const int x_mv = il ? 4 : 0;
const int gh_mv = il ? 12 : 0;
const int gh_bk = il ? 0 : 4;
for (int i = 0; i < 8; i++) {
// extract the 5-th bits for x0 and x1
const uint8_t xh_0 = ((qh >> (gh_mv + 2*i )) << gh_bk) & 0x10;
const uint8_t xh_1 = ((qh >> (gh_mv + 2*i+1)) << gh_bk) & 0x10;
// combine the 4-bits from qs with the 5th bit
const int32_t x0 = ((((qs[i] ) & mask) >> x_mv) | xh_0);
const int32_t x1 = ((((qs[i] >> 8) & mask) >> x_mv) | xh_1);
reg[i/2][2*(i%2)+0] = d * x0 + m;
reg[i/2][2*(i%2)+1] = d * x1 + m;
}
}
template
void dequantize_q8_0(device const block_q8_0 *xb, short il, thread type4x4 & reg) {
device const int8_t * qs = ((device const int8_t *)xb->qs);
const half d = xb->d;
for (int i = 0; i < 16; i++) {
reg[i/4][i%4] = (qs[i + 16*il] * d);
}
}
template
void dequantize_q2_K(device const block_q2_K *xb, short il, thread type4x4 & reg) {
const float d = xb->d;
const float min = xb->dmin;
device const uint8_t * q = (device const uint8_t *)xb->qs;
float dl, ml;
uint8_t sc = xb->scales[il];
q = q + 32*(il/8) + 16*(il&1);
il = (il/2)%4;
half coef = il>1 ? (il>2 ? 1/64.h : 1/16.h) : (il>0 ? 1/4.h : 1.h);
uchar mask = il>1 ? (il>2 ? 192 : 48) : (il>0 ? 12 : 3);
dl = d * (sc & 0xF) * coef, ml = min * (sc >> 4);
for (int i = 0; i < 16; ++i) {
reg[i/4][i%4] = dl * (q[i] & mask) - ml;
}
}
template
void dequantize_q3_K(device const block_q3_K *xb, short il, thread type4x4 & reg) {
const half d_all = xb->d;
device const uint8_t * q = (device const uint8_t *)xb->qs;
device const uint8_t * h = (device const uint8_t *)xb->hmask;
device const int8_t * scales = (device const int8_t *)xb->scales;
q = q + 32 * (il/8) + 16 * (il&1);
h = h + 16 * (il&1);
uint8_t m = 1 << (il/2);
uint16_t kmask1 = (il/4)>1 ? ((il/4)>2 ? 192 : 48) : \
((il/4)>0 ? 12 : 3);
uint16_t kmask2 = il/8 ? 0xF0 : 0x0F;
uint16_t scale_2 = scales[il%8], scale_1 = scales[8 + il%4];
int16_t dl_int = (il/4)&1 ? (scale_2&kmask2) | ((scale_1&kmask1) << 2)
: (scale_2&kmask2) | ((scale_1&kmask1) << 4);
float dl = il<8 ? d_all * (dl_int - 32.f) : d_all * (dl_int / 16.f - 32.f);
const float ml = 4.f * dl;
il = (il/2) & 3;
const half coef = il>1 ? (il>2 ? 1/64.h : 1/16.h) : (il>0 ? 1/4.h : 1.h);
const uint8_t mask = il>1 ? (il>2 ? 192 : 48) : (il>0 ? 12 : 3);
dl *= coef;
for (int i = 0; i < 16; ++i) {
reg[i/4][i%4] = dl * (q[i] & mask) - (h[i] & m ? 0 : ml);
}
}
static inline uchar2 get_scale_min_k4_just2(int j, int k, device const uchar * q) {
return j < 4 ? uchar2{uchar(q[j+0+k] & 63), uchar(q[j+4+k] & 63)}
: uchar2{uchar((q[j+4+k] & 0xF) | ((q[j-4+k] & 0xc0) >> 2)), uchar((q[j+4+k] >> 4) | ((q[j-0+k] & 0xc0) >> 2))};
}
template
void dequantize_q4_K(device const block_q4_K *xb, short il, thread type4x4 & reg) {
device const uchar * q = xb->qs;
short is = (il/4) * 2;
q = q + (il/4) * 32 + 16 * (il&1);
il = il & 3;
const uchar2 sc = get_scale_min_k4_just2(is, il/2, xb->scales);
const float d = il < 2 ? xb->d : xb->d / 16.h;
const float min = xb->dmin;
const float dl = d * sc[0];
const float ml = min * sc[1];
const ushort mask = il<2 ? 0x0F : 0xF0;
for (int i = 0; i < 16; ++i) {
reg[i/4][i%4] = dl * (q[i] & mask) - ml;
}
}
template
void dequantize_q5_K(device const block_q5_K *xb, short il, thread type4x4 & reg) {
device const uint8_t * q = xb->qs;
device const uint8_t * qh = xb->qh;
short is = (il/4) * 2;
q = q + 32 * (il/4) + 16 * (il&1);
qh = qh + 16 * (il&1);
uint8_t ul = 1 << (il/2);
il = il & 3;
const uchar2 sc = get_scale_min_k4_just2(is, il/2, xb->scales);
const float d = il < 2 ? xb->d : xb->d / 16.f;
const float min = xb->dmin;
const float dl = d * sc[0];
const float ml = min * sc[1];
const ushort mask = il<2 ? 0x0F : 0xF0;
const float qh_val = il<2 ? 16.f : 256.f;
for (int i = 0; i < 16; ++i) {
reg[i/4][i%4] = dl * ((q[i] & mask) + (qh[i] & ul ? qh_val : 0)) - ml;
}
}
template
void dequantize_q6_K(device const block_q6_K *xb, short il, thread type4x4 & reg) {
const half d_all = xb->d;
device const uint8_t * ql = (device const uint8_t *)xb->ql;
device const uint8_t * qh = (device const uint8_t *)xb->qh;
device const int8_t * scales = (device const int8_t *)xb->scales;
ql = ql + 64*(il/8) + 32*((il/2)&1) + 16*(il&1);
qh = qh + 32*(il/8) + 16*(il&1);
float sc = scales[(il%2) + 2 * ((il/2))];
il = (il/2) & 3;
const uint16_t kmask1 = il>1 ? (il>2 ? 192 : 48) : (il>0 ? 12 : 3);
const uint16_t kmask2 = il>1 ? 0xF0 : 0x0F;
const float coef = il>1 ? 1.f/16.f : 1.f;
const float ml = d_all * sc * 32.f;
const float dl = d_all * sc * coef;
for (int i = 0; i < 16; ++i) {
const half q = il&1 ? ((ql[i] & kmask2) | ((qh[i] & kmask1) << 2))
: ((ql[i] & kmask2) | ((qh[i] & kmask1) << 4));
reg[i/4][i%4] = dl * q - ml;
}
}
template
void dequantize_iq2_xxs(device const block_iq2_xxs * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const float d = xb->d;
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
// each block of 32 needs 2 uint32_t's for the quants & scale, so 4 uint16_t's.
device const uint16_t * q2 = xb->qs + 4*ib32;
const uint32_t aux32_g = q2[0] | (q2[1] << 16);
const uint32_t aux32_s = q2[2] | (q2[3] << 16);
thread const uint8_t * aux8 = (thread const uint8_t *)&aux32_g;
const float dl = d * (0.5f + (aux32_s >> 28)) * 0.25f;
constant uint8_t * grid = (constant uint8_t *)(iq2xxs_grid + aux8[2*il+0]);
uint8_t signs = ksigns_iq2xs[(aux32_s >> 14*il) & 127];
for (int i = 0; i < 8; ++i) {
reg[i/4][i%4] = dl * grid[i] * (signs & kmask_iq2xs[i] ? -1.f : 1.f);
}
grid = (constant uint8_t *)(iq2xxs_grid + aux8[2*il+1]);
signs = ksigns_iq2xs[(aux32_s >> (14*il+7)) & 127];
for (int i = 0; i < 8; ++i) {
reg[2+i/4][i%4] = dl * grid[i] * (signs & kmask_iq2xs[i] ? -1.f : 1.f);
}
}
template
void dequantize_iq2_xs(device const block_iq2_xs * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const float d = xb->d;
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
device const uint16_t * q2 = xb->qs + 4*ib32;
const float dl = d * (0.5f + ((xb->scales[ib32] >> 4*il) & 0xf)) * 0.25f;
constant uint8_t * grid = (constant uint8_t *)(iq2xs_grid + (q2[2*il+0] & 511));
uint8_t signs = ksigns_iq2xs[q2[2*il+0] >> 9];
for (int i = 0; i < 8; ++i) {
reg[i/4][i%4] = dl * grid[i] * (signs & kmask_iq2xs[i] ? -1.f : 1.f);
}
grid = (constant uint8_t *)(iq2xs_grid + (q2[2*il+1] & 511));
signs = ksigns_iq2xs[q2[2*il+1] >> 9];
for (int i = 0; i < 8; ++i) {
reg[2+i/4][i%4] = dl * grid[i] * (signs & kmask_iq2xs[i] ? -1.f : 1.f);
}
}
template
void dequantize_iq3_xxs(device const block_iq3_xxs * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const float d = xb->d;
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
device const uint8_t * q3 = xb->qs + 8*ib32;
device const uint16_t * gas = (device const uint16_t *)(xb->qs + QK_K/4) + 2*ib32;
const uint32_t aux32 = gas[0] | (gas[1] << 16);
const float dl = d * (0.5f + (aux32 >> 28)) * 0.5f;
constant uint8_t * grid1 = (constant uint8_t *)(iq3xxs_grid + q3[4*il+0]);
constant uint8_t * grid2 = (constant uint8_t *)(iq3xxs_grid + q3[4*il+1]);
uint8_t signs = ksigns_iq2xs[(aux32 >> 14*il) & 127];
for (int i = 0; i < 4; ++i) {
reg[0][i] = dl * grid1[i] * (signs & kmask_iq2xs[i+0] ? -1.f : 1.f);
reg[1][i] = dl * grid2[i] * (signs & kmask_iq2xs[i+4] ? -1.f : 1.f);
}
grid1 = (constant uint8_t *)(iq3xxs_grid + q3[4*il+2]);
grid2 = (constant uint8_t *)(iq3xxs_grid + q3[4*il+3]);
signs = ksigns_iq2xs[(aux32 >> (14*il+7)) & 127];
for (int i = 0; i < 4; ++i) {
reg[2][i] = dl * grid1[i] * (signs & kmask_iq2xs[i+0] ? -1.f : 1.f);
reg[3][i] = dl * grid2[i] * (signs & kmask_iq2xs[i+4] ? -1.f : 1.f);
}
}
template
void dequantize_iq3_s(device const block_iq3_s * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const float d = xb->d;
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
device const uint8_t * qs = xb->qs + 8*ib32;
device const uint8_t * signs = xb->signs + 4*ib32 + 2*il;
const uint8_t qh = xb->qh[ib32] >> 4*il;
const float dl = d * (1 + 2*((xb->scales[ib32/2] >> 4*(ib32%2)) & 0xf));
constant uint8_t * grid1 = (constant uint8_t *)(iq3s_grid + (qs[4*il+0] | ((qh << 8) & 256)));
constant uint8_t * grid2 = (constant uint8_t *)(iq3s_grid + (qs[4*il+1] | ((qh << 7) & 256)));
for (int i = 0; i < 4; ++i) {
reg[0][i] = dl * grid1[i] * select(1, -1, signs[0] & kmask_iq2xs[i+0]);
reg[1][i] = dl * grid2[i] * select(1, -1, signs[0] & kmask_iq2xs[i+4]);
}
grid1 = (constant uint8_t *)(iq3s_grid + (qs[4*il+2] | ((qh << 6) & 256)));
grid2 = (constant uint8_t *)(iq3s_grid + (qs[4*il+3] | ((qh << 5) & 256)));
for (int i = 0; i < 4; ++i) {
reg[2][i] = dl * grid1[i] * select(1, -1, signs[1] & kmask_iq2xs[i+0]);
reg[3][i] = dl * grid2[i] * select(1, -1, signs[1] & kmask_iq2xs[i+4]);
}
}
template
void dequantize_iq2_s(device const block_iq2_s * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const float d = xb->d;
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
device const uint8_t * qs = xb->qs + 4*ib32 + 2*il;
device const uint8_t * signs = qs + QK_K/8;
const uint8_t qh = xb->qh[ib32] >> 4*il;
const float dl = d * (0.5f + ((xb->scales[ib32] >> 4*il) & 0xf)) * 0.25f;
constant uint8_t * grid1 = (constant uint8_t *)(iq2s_grid + (qs[0] | ((qh << 8) & 0x300)));
constant uint8_t * grid2 = (constant uint8_t *)(iq2s_grid + (qs[1] | ((qh << 6) & 0x300)));
for (int i = 0; i < 8; ++i) {
reg[i/4+0][i%4] = dl * grid1[i] * select(1, -1, signs[0] & kmask_iq2xs[i]);
reg[i/4+2][i%4] = dl * grid2[i] * select(1, -1, signs[1] & kmask_iq2xs[i]);
}
}
template
void dequantize_iq1_s(device const block_iq1_s * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const int ib32 = il/2;
il = il%2;
const float d = xb->d;
device const uint8_t * qs = xb->qs + 4*ib32 + 2*il;
device const uint16_t * qh = xb->qh;
const float dl = d * (2*((qh[ib32] >> 12) & 7) + 1);
const float ml = dl * (qh[ib32] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA);
const uint16_t h = qh[ib32] >> 6*il;
constant uint8_t * grid1 = (constant uint8_t *)(iq1s_grid_gpu + (qs[0] | ((h << 8) & 0x700)));
constant uint8_t * grid2 = (constant uint8_t *)(iq1s_grid_gpu + (qs[1] | ((h << 5) & 0x700)));
for (int i = 0; i < 4; ++i) {
reg[0][i] = dl * (grid1[i] & 0xf) + ml;
reg[1][i] = dl * (grid1[i] >> 4) + ml;
reg[2][i] = dl * (grid2[i] & 0xf) + ml;
reg[3][i] = dl * (grid2[i] >> 4) + ml;
}
}
template
void dequantize_iq1_m(device const block_iq1_m * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const int ib32 = il/2;
il = il%2;
device const uint16_t * sc = (device const uint16_t *)xb->scales;
iq1m_scale_t scale;
scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00f0) | ((sc[2] >> 4) & 0x0f00) | (sc[3] & 0xf000);
const float d = scale.f16;
device const uint8_t * qs = xb->qs + 4*ib32 + 2*il;
device const uint8_t * qh = xb->qh + 2*ib32 + il;
const float dl = d * (2*((sc[ib32/2] >> (6*(ib32%2)+3*il)) & 7) + 1);
const float ml1 = dl * (qh[0] & 0x08 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA);
const float ml2 = dl * (qh[0] & 0x80 ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA);
constant uint8_t * grid1 = (constant uint8_t *)(iq1s_grid_gpu + (qs[0] | ((qh[0] << 8) & 0x700)));
constant uint8_t * grid2 = (constant uint8_t *)(iq1s_grid_gpu + (qs[1] | ((qh[0] << 4) & 0x700)));
for (int i = 0; i < 4; ++i) {
reg[0][i] = dl * (grid1[i] & 0xf) + ml1;
reg[1][i] = dl * (grid1[i] >> 4) + ml1;
reg[2][i] = dl * (grid2[i] & 0xf) + ml2;
reg[3][i] = dl * (grid2[i] >> 4) + ml2;
}
}
template
void dequantize_iq4_nl(device const block_iq4_nl * xb, short il, thread type4x4 & reg) {
device const uint16_t * q4 = (device const uint16_t *)xb->qs;
const float d = xb->d;
uint32_t aux32;
thread const uint8_t * q8 = (thread const uint8_t *)&aux32;
for (int i = 0; i < 4; ++i) {
aux32 = ((q4[2*i] | (q4[2*i+1] << 16)) >> 4*il) & 0x0f0f0f0f;
reg[i][0] = d * kvalues_iq4nl_f[q8[0]];
reg[i][1] = d * kvalues_iq4nl_f[q8[1]];
reg[i][2] = d * kvalues_iq4nl_f[q8[2]];
reg[i][3] = d * kvalues_iq4nl_f[q8[3]];
}
}
template
void dequantize_iq4_xs(device const block_iq4_xs * xb, short il, thread type4x4 & reg) {
// il is 0...15 for QK_K = 256 => index of block of 32 is il/2
const int ib32 = il/2;
il = il%2;
// il = 0 or 1. il = 0 processes the first 16 quants in a block of 32, il = 1 the second 16
device const uint32_t * q4 = (device const uint32_t *)xb->qs + 4*ib32;
const int ls = ((xb->scales_l[ib32/2] >> 4*(ib32%2)) & 0xf) | (((xb->scales_h >> 2*ib32) & 3) << 4);
const float d = (float)xb->d * (ls - 32);
uint32_t aux32;
thread const uint8_t * q8 = (thread const uint8_t *)&aux32;
for (int i = 0; i < 4; ++i) {
aux32 = (q4[i] >> 4*il) & 0x0f0f0f0f;
reg[i][0] = d * kvalues_iq4nl_f[q8[0]];
reg[i][1] = d * kvalues_iq4nl_f[q8[1]];
reg[i][2] = d * kvalues_iq4nl_f[q8[2]];
reg[i][3] = d * kvalues_iq4nl_f[q8[3]];
}
}
template
kernel void kernel_get_rows_q(
device const void * src0,
device const void * src1,
device float * dst,
constant int64_t & ne00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb1,
constant uint64_t & nb2,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint3 tptg [[threads_per_threadgroup]]) {
const int64_t i10 = tgpig.x;
const int64_t i11 = tgpig.y;
const int64_t r = ((const device int32_t *) ((const device char *) src1 + i11*nb11 + i10*nb10))[0];
const int64_t i02 = i11;
for (int64_t ind = tiitg; ind < ne00/16; ind += tptg.x) {
float4x4 temp;
dequantize_func(((device const block_q *) ((const device char *) src0 + r*nb01 + i02*nb02)) + ind/nl, ind%nl, temp);
*(((device float4x4 *) ((device char *) dst + i11*nb2 + i10*nb1)) + ind) = temp;
}
}
template
kernel void kernel_get_rows_f(
device const void * src0,
device const void * src1,
device float * dst,
constant int64_t & ne00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb1,
constant uint64_t & nb2,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint3 tptg [[threads_per_threadgroup]]) {
const int64_t i10 = tgpig.x;
const int64_t i11 = tgpig.y;
const int64_t r = ((const device int32_t *) ((const device char *) src1 + i11*nb11 + i10*nb10))[0];
const int64_t i02 = i11;
for (int ind = tiitg; ind < ne00; ind += tptg.x) {
(( device float *) (( device char *) dst + i11*nb2 + i10*nb1))[ind] =
((const device T *) ((const device char *) src0 + i02*nb02 + r*nb01))[ind];
}
}
kernel void kernel_get_rows_i32(
device const void * src0,
device const void * src1,
device int32_t * dst,
constant int64_t & ne00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb1,
constant uint64_t & nb2,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint3 tptg [[threads_per_threadgroup]]) {
const int64_t i10 = tgpig.x;
const int64_t i11 = tgpig.y;
const int64_t r = ((const device int32_t *) ((const device char *) src1 + i11*nb11 + i10*nb10))[0];
const int64_t i02 = i11;
for (int ind = tiitg; ind < ne00; ind += tptg.x) {
(( device int32_t *) (( device char *) dst + i11*nb2 + i10*nb1))[ind] =
((const device int32_t *) ((const device char *) src0 + i02*nb02 + r*nb01))[ind];
}
}
#define BLOCK_SIZE_M 64 // 8 simdgroup matrices from matrix A
#define BLOCK_SIZE_N 32 // 4 simdgroup matrices from matrix B
#define BLOCK_SIZE_K 32
#define THREAD_MAT_M 4 // each thread take 4 simdgroup matrices from matrix A
#define THREAD_MAT_N 2 // each thread take 2 simdgroup matrices from matrix B
#define THREAD_PER_BLOCK 128
#define THREAD_PER_ROW 2 // 2 thread for each row in matrix A to load numbers
#define THREAD_PER_COL 4 // 4 thread for each row in matrix B to load numbers
#define SG_MAT_SIZE 64 // simdgroup matrix is of shape 8x8
#define SG_MAT_ROW 8
// each block_q contains 16*nl weights
template
kernel void kernel_mul_mm(device const uchar * src0,
device const uchar * src1,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne02,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint & r2,
constant uint & r3,
threadgroup uchar * shared_memory [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
threadgroup T * sa = (threadgroup T *)(shared_memory);
threadgroup float * sb = (threadgroup float *)(shared_memory + 4096);
const uint r0 = tgpig.y;
const uint r1 = tgpig.x;
const uint im = tgpig.z;
// if this block is of 64x32 shape or smaller
short n_rows = (ne0 - r0 * BLOCK_SIZE_M < BLOCK_SIZE_M) ? (ne0 - r0 * BLOCK_SIZE_M) : BLOCK_SIZE_M;
short n_cols = (ne1 - r1 * BLOCK_SIZE_N < BLOCK_SIZE_N) ? (ne1 - r1 * BLOCK_SIZE_N) : BLOCK_SIZE_N;
// a thread shouldn't load data outside of the matrix
short thread_row = ((short)tiitg/THREAD_PER_ROW) < n_rows ? ((short)tiitg/THREAD_PER_ROW) : n_rows - 1;
short thread_col = ((short)tiitg/THREAD_PER_COL) < n_cols ? ((short)tiitg/THREAD_PER_COL) : n_cols - 1;
simdgroup_T8x8 ma[4];
simdgroup_float8x8 mb[2];
simdgroup_float8x8 c_res[8];
for (int i = 0; i < 8; i++){
c_res[i] = make_filled_simdgroup_matrix(0.f);
}
short il = (tiitg % THREAD_PER_ROW);
const uint i12 = im%ne12;
const uint i13 = im/ne12;
uint offset0 = (i12/r2)*nb02 + (i13/r3)*(nb02*ne02);
ushort offset1 = il/nl;
device const block_q * x = (device const block_q *)(src0 + (r0 * BLOCK_SIZE_M + thread_row) * nb01 + offset0) + offset1;
device const float * y = (device const float *)(src1
+ nb12 * im
+ nb11 * (r1 * BLOCK_SIZE_N + thread_col)
+ nb10 * (BLOCK_SIZE_K / THREAD_PER_COL * (tiitg % THREAD_PER_COL)));
for (int loop_k = 0; loop_k < ne00; loop_k += BLOCK_SIZE_K) {
// load data and store to threadgroup memory
T4x4 temp_a;
dequantize_func(x, il, temp_a);
threadgroup_barrier(mem_flags::mem_threadgroup);
#pragma unroll(16)
for (int i = 0; i < 16; i++) {
*(sa + SG_MAT_SIZE * ((tiitg / THREAD_PER_ROW / 8) \
+ (tiitg % THREAD_PER_ROW) * 16 + (i / 8) * 8) \
+ (tiitg / THREAD_PER_ROW) % 8 + (i & 7) * 8) = temp_a[i/4][i%4];
}
*(threadgroup float2x4 *)(sb + (tiitg % THREAD_PER_COL) * 8 * 32 + 8 * (tiitg / THREAD_PER_COL)) = *((device float2x4 *)y);
il = (il + 2 < nl) ? il + 2 : il % 2;
x = (il < 2) ? x + (2+nl-1)/nl : x;
y += BLOCK_SIZE_K;
threadgroup_barrier(mem_flags::mem_threadgroup);
// load matrices from threadgroup memory and conduct outer products
threadgroup T * lsma = (sa + THREAD_MAT_M * SG_MAT_SIZE * (sgitg % 2));
threadgroup float * lsmb = (sb + THREAD_MAT_N * SG_MAT_SIZE * (sgitg / 2));
#pragma unroll(4)
for (int ik = 0; ik < BLOCK_SIZE_K / 8; ik++) {
#pragma unroll(4)
for (int i = 0; i < 4; i++) {
simdgroup_load(ma[i],lsma + SG_MAT_SIZE * i);
}
simdgroup_barrier(mem_flags::mem_none);
#pragma unroll(2)
for (int i = 0; i < 2; i++) {
simdgroup_load(mb[i],lsmb + SG_MAT_SIZE * i);
}
lsma += BLOCK_SIZE_M / SG_MAT_ROW * SG_MAT_SIZE;
lsmb += BLOCK_SIZE_N / SG_MAT_ROW * SG_MAT_SIZE;
#pragma unroll(8)
for (int i = 0; i < 8; i++){
simdgroup_multiply_accumulate(c_res[i], mb[i/4], ma[i%4], c_res[i]);
}
}
}
if ((r0 + 1) * BLOCK_SIZE_M <= ne0 && (r1 + 1) * BLOCK_SIZE_N <= ne1) {
device float * C = dst + (BLOCK_SIZE_M * r0 + 32 * (sgitg & 1)) \
+ (BLOCK_SIZE_N * r1 + 16 * (sgitg >> 1)) * ne0 + im*ne1*ne0;
for (int i = 0; i < 8; i++) {
simdgroup_store(c_res[i], C + 8 * (i%4) + 8 * ne0 * (i/4), ne0);
}
} else {
// block is smaller than 64x32, we should avoid writing data outside of the matrix
threadgroup_barrier(mem_flags::mem_threadgroup);
threadgroup float * temp_str = ((threadgroup float *)shared_memory) \
+ 32 * (sgitg&1) + (16 * (sgitg>>1)) * BLOCK_SIZE_M;
for (int i = 0; i < 8; i++) {
simdgroup_store(c_res[i], temp_str + 8 * (i%4) + 8 * BLOCK_SIZE_M * (i/4), BLOCK_SIZE_M);
}
threadgroup_barrier(mem_flags::mem_threadgroup);
device float * C = dst + (BLOCK_SIZE_M * r0) + (BLOCK_SIZE_N * r1) * ne0 + im*ne1*ne0;
if (sgitg == 0) {
for (int i = 0; i < n_rows; i++) {
for (int j = tiitg; j < n_cols; j += BLOCK_SIZE_N) {
*(C + i + j * ne0) = *(temp_str + i + j * BLOCK_SIZE_M);
}
}
}
}
}
// same as kernel_mul_mm_impl, but src1 and dst are accessed via indices stored in rowids
template
void kernel_mul_mm_id_impl(
device const uchar * src0,
device const uchar * src1,
threadgroup ushort2 * rowids,
device float * dst,
constant int64_t & ne00,
constant int64_t & ne02,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne11,
constant int64_t & ne12,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
int64_t ne1,
int64_t ne0ne1,
threadgroup uchar * shared_memory,
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
threadgroup half * sa = (threadgroup half *)(shared_memory);
threadgroup float * sb = (threadgroup float *)(shared_memory + 4096);
const uint r0 = tgpig.y;
const uint r1 = tgpig.x;
if (r1 * BLOCK_SIZE_N >= ne1) return;
// if this block is of 64x32 shape or smaller
short n_rows = (ne0 - r0 * BLOCK_SIZE_M < BLOCK_SIZE_M) ? (ne0 - r0 * BLOCK_SIZE_M) : BLOCK_SIZE_M;
short n_cols = (ne1 - r1 * BLOCK_SIZE_N < BLOCK_SIZE_N) ? (ne1 - r1 * BLOCK_SIZE_N) : BLOCK_SIZE_N;
// a thread shouldn't load data outside of the matrix
short thread_row = ((short)tiitg/THREAD_PER_ROW) < n_rows ? ((short)tiitg/THREAD_PER_ROW) : n_rows - 1;
short thread_col = ((short)tiitg/THREAD_PER_COL) < n_cols ? ((short)tiitg/THREAD_PER_COL) : n_cols - 1;
simdgroup_half8x8 ma[4];
simdgroup_float8x8 mb[2];
simdgroup_float8x8 c_res[8];
for (int i = 0; i < 8; i++){
c_res[i] = make_filled_simdgroup_matrix(0.f);
}
short il = (tiitg % THREAD_PER_ROW);
ushort offset1 = il/nl;
threadgroup const auto & id = rowids[r1 * BLOCK_SIZE_N + thread_col];
device const block_q * x = (device const block_q *)(src0 + (r0 * BLOCK_SIZE_M + thread_row) * nb01) + offset1;
device const float * y = (device const float *)(src1
+ nb12 * id[1]
+ nb11 * (id[0] % ne11)
+ nb10 * (BLOCK_SIZE_K / THREAD_PER_COL * (tiitg % THREAD_PER_COL)));
for (int loop_k = 0; loop_k < ne00; loop_k += BLOCK_SIZE_K) {
// load data and store to threadgroup memory
half4x4 temp_a;
dequantize_func(x, il, temp_a);
threadgroup_barrier(mem_flags::mem_threadgroup);
for (int i = 0; i < 16; i++) {
*(sa + SG_MAT_SIZE * ((tiitg / THREAD_PER_ROW / 8) \
+ (tiitg % THREAD_PER_ROW) * 16 + (i / 8) * 8) \
+ (tiitg / THREAD_PER_ROW) % 8 + (i & 7) * 8) = temp_a[i/4][i%4];
}
*(threadgroup float2x4 *)(sb + (tiitg % THREAD_PER_COL) * 8 * 32 + 8 * (tiitg / THREAD_PER_COL)) = *((device float2x4 *)y);
il = (il + 2 < nl) ? il + 2 : il % 2;
x = (il < 2) ? x + (2+nl-1)/nl : x;
y += BLOCK_SIZE_K;
threadgroup_barrier(mem_flags::mem_threadgroup);
// load matrices from threadgroup memory and conduct outer products
threadgroup half * lsma = (sa + THREAD_MAT_M * SG_MAT_SIZE * (sgitg % 2));
threadgroup float * lsmb = (sb + THREAD_MAT_N * SG_MAT_SIZE * (sgitg / 2));
for (int ik = 0; ik < BLOCK_SIZE_K / 8; ik++) {
for (int i = 0; i < 4; i++) {
simdgroup_load(ma[i], lsma + SG_MAT_SIZE * i);
}
simdgroup_barrier(mem_flags::mem_none);
for (int i = 0; i < 2; i++) {
simdgroup_load(mb[i], lsmb + SG_MAT_SIZE * i);
}
lsma += BLOCK_SIZE_M / SG_MAT_ROW * SG_MAT_SIZE;
lsmb += BLOCK_SIZE_N / SG_MAT_ROW * SG_MAT_SIZE;
for (int i = 0; i < 8; i++){
simdgroup_multiply_accumulate(c_res[i], mb[i/4], ma[i%4], c_res[i]);
}
}
}
{
threadgroup_barrier(mem_flags::mem_threadgroup);
threadgroup float * temp_str = ((threadgroup float *)shared_memory) \
+ 32 * (sgitg&1) + (16 * (sgitg>>1)) * BLOCK_SIZE_M;
for (int i = 0; i < 8; i++) {
simdgroup_store(c_res[i], temp_str + 8 * (i%4) + 8 * BLOCK_SIZE_M * (i/4), BLOCK_SIZE_M);
}
threadgroup_barrier(mem_flags::mem_threadgroup);
device float * C = dst + (BLOCK_SIZE_M * r0);
if (sgitg == 0) {
for (int j = tiitg; j < n_cols; j += BLOCK_SIZE_N) {
threadgroup const auto & jid = rowids[r1 * BLOCK_SIZE_N + j];
int joff = jid[0] * ne0 + jid[1] * ne0ne1;
for (int i = 0; i < n_rows; i++) {
*(C + i + joff) = *(temp_str + i + j * BLOCK_SIZE_M);
}
}
}
}
}
template
kernel void kernel_mul_mm_id(
device const uchar * src0s,
device const uchar * src1,
device float * dst,
device const uchar * ids,
constant int64_t & nei0,
constant int64_t & nei1,
constant uint64_t & nbi1,
constant int64_t & ne00,
constant int64_t & ne02,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint64_t & nb1,
threadgroup uchar * shared_memory [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
const int32_t i02 = tgpig.z;
tgpig.z = 0;
device const uchar * src0 = src0s + i02*nb02;
// row indices
threadgroup ushort2 * rowids = (threadgroup ushort2 *)(shared_memory + 8192);
// TODO: parallelize this loop
int64_t _ne1 = 0;
for (ushort ii1 = 0; ii1 < nei1; ii1++) {
for (ushort ii0 = 0; ii0 < nei0; ii0++) {
int32_t id = ((device int32_t *) (ids + ii1*nbi1))[ii0];
if (id == i02) {
//if (tiitg == 0) {
rowids[_ne1] = ushort2(ii0, ii1);
//}
_ne1++;
}
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
kernel_mul_mm_id_impl(
src0,
src1,
rowids,
dst,
ne00,
ne02,
nb01,
nb02,
ne11,
ne12,
nb10,
nb11,
nb12,
ne0,
_ne1,
ne0*ne1,
shared_memory,
tgpig,
tiitg,
sgitg);
}
#define QK_NL 16
//
// get rows
//
typedef decltype(kernel_get_rows_f) get_rows_f_t;
template [[host_name("kernel_get_rows_f32")]] kernel get_rows_f_t kernel_get_rows_f;
template [[host_name("kernel_get_rows_f16")]] kernel get_rows_f_t kernel_get_rows_f;
typedef decltype(kernel_get_rows_q) get_rows_q_t;
template [[host_name("kernel_get_rows_q4_0")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q4_1")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q5_0")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q5_1")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q8_0")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q2_K")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q3_K")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q4_K")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q5_K")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_q6_K")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq2_xxs")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq2_xs")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq3_xxs")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq3_s")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq2_s")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq1_s")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq1_m")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq4_nl")]] kernel get_rows_q_t kernel_get_rows_q;
template [[host_name("kernel_get_rows_iq4_xs")]] kernel get_rows_q_t kernel_get_rows_q;
//
// matrix-matrix multiplication
//
typedef decltype(kernel_mul_mm) mat_mm_t;
template [[host_name("kernel_mul_mm_f32_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_f16_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q4_0_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q4_1_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q5_0_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q5_1_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q8_0_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q2_K_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q3_K_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q4_K_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q5_K_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_q6_K_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq2_xxs_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq2_xs_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq3_xxs_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq3_s_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq2_s_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq1_s_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq1_m_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq4_nl_f32")]] kernel mat_mm_t kernel_mul_mm;
template [[host_name("kernel_mul_mm_iq4_xs_f32")]] kernel mat_mm_t kernel_mul_mm;
//
// indirect matrix-matrix multiplication
//
typedef decltype(kernel_mul_mm_id) mat_mm_id_t;
template [[host_name("kernel_mul_mm_id_f32_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_f16_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q4_0_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q4_1_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q5_0_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q5_1_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q8_0_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q2_K_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q3_K_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q4_K_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q5_K_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_q6_K_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq2_xxs_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq2_xs_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq3_xxs_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq3_s_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq2_s_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq1_s_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq1_m_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq4_nl_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
template [[host_name("kernel_mul_mm_id_iq4_xs_f32")]] kernel mat_mm_id_t kernel_mul_mm_id;
//
// matrix-vector multiplication
//
typedef void (kernel_mul_mv_impl_t)(
device const char * src0,
device const char * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
uint64_t nb00,
uint64_t nb01,
uint64_t nb02,
int64_t ne10,
int64_t ne11,
int64_t ne12,
uint64_t nb10,
uint64_t nb11,
uint64_t nb12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
uint3 tgpig,
uint tiisg);
typedef void (kernel_mul_mv2_impl_t)(
device const void * src0,
device const float * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
int64_t ne10,
int64_t ne12,
int64_t ne0,
int64_t ne1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiisg,
uint sgitg);
template
void mmv_fn(
device const char * src0,
device const char * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
uint64_t nb00,
uint64_t nb01,
uint64_t nb02,
int64_t ne10,
int64_t ne11,
int64_t ne12,
int64_t ne13,
uint64_t nb10,
uint64_t nb11,
uint64_t nb12,
int64_t ne0,
int64_t ne1,
uint64_t nb1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiitg,
uint tiisg,
uint sgitg) {
impl_fn(src0,src1,dst,ne00,ne01,ne02,nb00,nb01,nb02,ne10,ne11,ne12,nb10,nb11,nb12,ne0,ne1,r2,r3,tgpig,tiisg);
}
template
void mmv_fn(
device const char * src0,
device const char * src1,
device float * dst,
int64_t ne00,
int64_t ne01,
int64_t ne02,
uint64_t nb00,
uint64_t nb01,
uint64_t nb02,
int64_t ne10,
int64_t ne11,
int64_t ne12,
int64_t ne13,
uint64_t nb10,
uint64_t nb11,
uint64_t nb12,
int64_t ne0,
int64_t ne1,
uint64_t nb1,
uint r2,
uint r3,
threadgroup int8_t * shared_values,
uint3 tgpig,
uint tiitg,
uint tiisg,
uint sgitg) {
impl_fn(src0,(const device float *)src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,shared_values,tgpig,tiisg,sgitg);
}
typedef decltype(mmv_fn>) mul_mv_impl_fn_t;
template
kernel void kernel_mul_mv_id(
device const char * src0s,
device const char * src1,
device float * dst,
device const char * ids,
constant int64_t & nei0,
constant int64_t & nei1,
constant uint64_t & nbi1,
constant int64_t & ne00,
constant int64_t & ne01,
constant int64_t & ne02,
constant uint64_t & nb00,
constant uint64_t & nb01,
constant uint64_t & nb02,
constant int64_t & ne10,
constant int64_t & ne11,
constant int64_t & ne12,
constant int64_t & ne13,
constant uint64_t & nb10,
constant uint64_t & nb11,
constant uint64_t & nb12,
constant int64_t & ne0,
constant int64_t & ne1,
constant uint64_t & nb1,
threadgroup int8_t * shared_values [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint tiitg[[thread_index_in_threadgroup]],
uint tiisg[[thread_index_in_simdgroup]],
uint sgitg[[simdgroup_index_in_threadgroup]]) {
const int iid1 = tgpig.z/nei0;
const int idx = tgpig.z%nei0;
tgpig.z = 0;
const int32_t i02 = ((device const int32_t *) (ids + iid1*nbi1))[idx];
const int64_t i11 = idx % ne11;
const int64_t i12 = iid1;
const int64_t i1 = idx;
const int64_t i2 = i12;
device const char * src0_cur = src0s + i02*nb02;
device const char * src1_cur = src1 + i11*nb11 + i12*nb12;
device float * dst_cur = dst + i1*ne0 + i2*ne1*ne0;
impl_fn(
/* src0 */ src0_cur,
/* src1 */ src1_cur,
/* dst */ dst_cur,
/* ne00 */ ne00,
/* ne01 */ ne01,
/* ne02 */ 1,//ne02,
/* nb00 */ nb00,
/* nb01 */ nb01,
/* nb02 */ nb02,
/* ne10 */ ne10,
/* ne11 */ 1,//ne11,
/* ne12 */ 1,//ne12,
/* ne13 */ 1,//ne13,
/* nb10 */ nb10,
/* nb11 */ nb11,
/* nb12 */ nb12,
/* ne0 */ ne0,
/* ne1 */ 1,//ne1,
/* nb1 */ nb1,
/* r2 */ 1,
/* r3 */ 1,
shared_values,
tgpig,
tiitg,
tiisg,
sgitg);
}
typedef decltype(kernel_mul_mv_id>>) kernel_mul_mv_id_t;
template [[host_name("kernel_mul_mv_id_f32_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_f16_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_q8_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_q4_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_q4_1_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_q5_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_q5_1_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>>;
template [[host_name("kernel_mul_mv_id_q2_K_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_q3_K_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_q4_K_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_q5_K_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_q6_K_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq1_s_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq1_m_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq2_xxs_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq2_xs_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq3_xxs_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq3_s_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq2_s_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq4_nl_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
template [[host_name("kernel_mul_mv_id_iq4_xs_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id>;
!?p(p@p(p SAp(p SAp(p SAp(ppppp SAp(p SAp(p UEp(RGp SAp(p SAp(p SAp(p SAp(p SAp(p(p@p pp(p pp(p SAp(p pp(p SAp(p SAp(p SAp(p SAp(?pp pp(p SAp(p`SLSLSDSAp(p SAp(`QA`QA`DRAREppUARFpSAUAWASDSHSEUHSDUA??pQ" `zCp@p(`1ApPRCpTETApSBT @_OBJC_CLASS_$_NSBundle Qr? ?@_OBJC_CLASS_$_NSString ??@_OBJC_CLASS_$_MTLCaptureDescriptor ??@_OBJC_CLASS_$_MTLCaptureManager ???????????@_OBJC_CLASS_$_MTLCompileOptions ???????????@__NSConcreteStackBlock q ?@___stack_chk_guard ?X?@___stderrp ?@___stdoutp ?@__dispatch_queue_attr_concurrent ?@_mach_task_self_ ??@dyld_stub_binder ??@__ZNSt12length_errorD1Ev ???????????@__ZNSt20bad_array_new_lengthD1Ev ?@__ZTINSt3__117__assoc_sub_stateE ??4?@__ZTISt12length_error ???????????@__ZTISt20bad_array_new_length ?@__ZTVN10__cxxabiv120__si_class_type_infoE ??4`?@__ZTVNSt3__117__assoc_sub_stateE ??????????` ?@__ZTVSt12length_error ?@__ZdaPv ?@__ZdlPv ?@__Znam ?@__Znwm ?@___gxx_personality_v0 ?@_OBJC_CLASS_$_NSMutableDictionary r? ?@_OBJC_CLASS_$_NSURL ??@___CFConstantStringClassReference q?5???@_OBJC_CLASS_$_NSObject r? ?@_OBJC_METACLASS_$_NSObject ????????????@__objc_empty_cache ??@_objc_msgSend q??@_objc_release ? @__ZTISt12length_error Qq?@__ZdaPv ??@__ZdlPv ?@__Znam ?@__Znwm ? r @_MTLCopyAllDevices ? r@_MTLCreateSystemDefaultDevice ? r@__Unwind_Resume ? r@__ZNSt11logic_errorC2EPKc ? r @__ZNSt13exception_ptrD1Ev ? r(@__ZNSt20bad_array_new_lengthC1Ev ? r0@__ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEmc ? r8@__ZNSt3__114__shared_countD2Ev ? r@@__ZNSt3__115__thread_structC1Ev ? rH@__ZNSt3__115__thread_structD1Ev ? rP@__ZNSt3__117__assoc_sub_state13set_exceptionESt13exception_ptr ? rX@__ZNSt3__117__assoc_sub_state16__on_zero_sharedEv ? r`@__ZNSt3__117__assoc_sub_state4waitEv ? rh@__ZNSt3__117__assoc_sub_state9set_valueEv ? rp@__ZNSt3__118condition_variableD1Ev ? rx@__ZNSt3__119__thread_local_dataEv ? r?@__ZNSt3__120__throw_system_errorEiPKc ? r?@__ZNSt3__15mutexD1Ev ? r?@__ZNSt3__16futureIvE3getEv ? r?@__ZNSt3__16futureIvEC1EPNS_17__assoc_sub_stateE ? r?@__ZNSt3__16futureIvED1Ev ? r?@__ZNSt3__16thread6detachEv ? r?@__ZNSt3__16threadD1Ev ? r?@__ZSt17current_exceptionv ? r?@__ZSt9terminatev ? r?@____chkstk_darwin ? r?@___bzero ? r?@___cxa_allocate_exception ? r?@___cxa_atexit ? r?@___cxa_begin_catch ? r?@___cxa_end_catch ? r?@___cxa_free_exception ? r?@___cxa_guard_abort ? r?@___cxa_guard_acquire ? r?@___cxa_guard_release ? r?@___cxa_rethrow ? r?@___cxa_throw ? r?@___error ? r?@___sincosf_stret ? r?@___stack_chk_fail ? r?@_abort ? r?@_calloc ? r?A_cblas_sgemm$NEWLAPACK$ILP64 ? r?@_clock ? r?@_clock_gettime ? r?@_cosf ? r?@_dispatch_apply ? r?@_dispatch_queue_create ? r?@_dispatch_release ? r?@_exp2f ? r?@_expf ? r?@_expm1f ? r?@_fclose ? r?@_fflush ? r?@_fopen$DARWIN_EXTSN ? r?@_fprintf ? r?@_fputc ? r?@_fputs ? r?@_fread ? r?@_free ? r?@_fseek ? r?@_ftell ? r?@_fwrite ? r?@_getenv ? r?@_log1pf ? r?@_log2 ? r?@_log2f ? r?@_logf ? r?@_lroundf ? r?@_malloc ? r?@_memcmp ? r?@_memcpy ? r?@_memset ? r?@_memset_pattern16 ? r?@_objc_alloc ? r?@_objc_autoreleasePoolPop ? r?@_objc_autoreleasePoolPush ? r?@_objc_enumerationMutation ? r?@_objc_opt_class ? r?@_objc_opt_new ? r?@_posix_memalign ? r?@_powf ? r?@_printf ? r?@_pthread_cond_broadcast ? r?@_pthread_cond_destroy ? r?@_pthread_cond_init ? r?@_pthread_cond_wait ? r?@_pthread_create ? r?@_pthread_join ? r?@_pthread_mutex_destroy ? r?@_pthread_mutex_init ? r?@_pthread_mutex_lock ? r?@_pthread_mutex_unlock ? r?@_pthread_self ? r?@_pthread_setschedparam ? r?@_pthread_setspecific ? r?@_putchar ? r?@_puts ? r?@_qsort ? r?@_realloc ? r?@_sched_yield ? r?@_sinf ? r?@_snprintf ? r?@_strcmp ? r?@_strdup ? r?@_strerror ? r?@_strlen ? r?@_strncpy ? r?@_sysconf ? r?@_sysctlbyname ? r?@_tanhf ? r?@_vDSP_maxv ? r?@_vDSP_sve ? r?@_vDSP_vadd ? r?@_vDSP_vdiv ? r?@_vDSP_vmul ? r?@_vDSP_vsmul ? r?@_vDSP_vsub ? r?@_vfprintf ? r?@_vm_allocate ? r?@_vm_deallocate ? r?@_vsnprintf ? _ ?X Ќ5 ??5 CLASS_$_GGMLMetalClass METACLASS_$_GGMLMetalClass ??# п# m Ks Q Ы# ??# ?# s exs k s _x q ?# ??# s ?xxs ? ??# ??# nl ?xs ? 1_ W2_ z3_ ?4_ ? ?? ?? Н ?? ? 0 ?1 ?K ? ?? ?? ?? 0 ?1 ?K ? ?? ?? ??# 0 ?K ? 2_K ?3_K ?4_ ?5_ ?6_K ?8_ ? ??" ??# 1_0 ?2_0 ? iq ?q ?tq ? ?y ?? ??_inplace ? ort ?s ? ? ??_inplace ? ?? ??_inplace ? ?? ?? ?? ??_inplace ? cast ?inplace ?rel_pos ? О1 ?_ ? ?? П ?? hape ?tride ? ?? ?? max ?sort ? ange ?e_same_s ?g ? b ?cc ?dd ?r ? ?? ?? ??_from_buft ? buffer ?ctx_tensors ? ??- Ш- init ?set_n_threads ? ?? ?? lear ?opy_tensor ? ?? ?? ?? ignment ?loc_size ? ?? ?? ?? ?? ?? al ?base ?max_size ?size ?type ?usage ? ?? ??_tensor ? ?? ?? host ?multi_buffer ? nit ?s_ ? ?? ?? ?? c ?free ?get_ ?i ?name ?reset ?set_usage ? ?? ?? ?? ignment ?loc_size ? ?? ?? al ?device ?max_size ? ?? ?? alloc_buffer ?get_ ?is_host ?name ? fer_ ?t_ ? las_ ?uf ? ?? ? ?? from_ptr ? type ? ?? ?? ?? ?? ?? abort_callback ? n_threads ? threadpool ? buffer_ ? init ? reg ? set_ ? ompare_graph_backend ? pu_ ?
?? ?? ?? from_host_ptr ?
type ?
?? Џ name ? type ? ackend_reg ?
uffer_ ?
y_ ? ?? ?? ?? ??_props ? ?? ?? ?? ?? ?? ?? ?? buft ? op ? ?? b ? count ? description ? get ? host_buffer_type ? init ? memory ? name ? offload_op ? supports_ ? type ? Ё _ ? ice_register ?
?? ?? ?? ?? ?? free ?
new ?
record ?
synchronize ?
wait ?
?? ?? ?? ?? fault_buffer_type ?
vice ? ?? alignment ?
de ?max_size ? ?? ??_async ? ?? ?_free ? mpute ?py ? ?? ?? ompute ?reate ? ?? c ?free ? co ?plan_ ? ?? et_ ?raph_ ?uid ? ?? ?? ?? name ?type ? est ?y_ ? ??- ?? ??* blas ?cpu ?metal ? nit_b ?s_ ? ??) ??) from_ptr ?type ? ??* ??) ??* ??* et_abort_callback ?upports_family ? buffer_ ?capture_next_compute ?init ?s ? ?? ?? alloc_buffer ?set_usage ? etal_ ?ulti_buffer_ ? ?? ?? ?? ?? ?? ?? count ?get ? ?? ??_proc_address ? ??* ?? by_name ?count ?dev_ ?get ?metal_init ?name ? ?? _ ?ister ? ? ? У ?? ackend ?uffer_size ? ?? ?? ?? backends ?copies ?splits ? ?? b ?n_ ?tensor_backend ? ? ??_async ? et_ ?raph_compute ? Т ?? ? rve ?t ? ?? ? eval_callback ?tensor_backend ? ?? et_ ?ynchronize ? alloc_graph ?free ?g ?new ?rese ?s ? ?? ?? buft ?op ? ?? ched_ ?upports_ ?ynchronize ? ? ?? ??_async ? ?? ??_async ? ?? ?? ??_async ? alloc ?copy ?get ?memset ?set ? ?? alloc_ ?b ? c ?
dev ?
event_ ?
free ?
g ?i ?m ?name ?offload_op ?reg ?s ?tensor_ ?view_init ? ?? ?_row ? ?? ?? ?? expand ?gradient_checkpointing ? ?? ?? backward_ ?forward_expand ?opt_adamw ? ackend_ ?f16_to_fp32 ?lck_size ?uild_ ? Р ?? n_repeat ?st ? ?? ?? ?? ?? ?? ?? 1d ?2d ?3d ?4d ? ??_ ? ?? ??_ph ? ?? ?? 1_ph ?k_p0 ? ??_s ? ?? ?? ?? 1d ?2d_p0 ? 1d ?2d ?depthwise_2d ?transpose_ ? cat ?t ?v_ ? ?? ?_inplace ? ?? n ?s ?unt_equal ? ?? ?? ?? ?? ?? ?? bmi ? nni ? bf16 ? v ? ??_ ? ?? ??2 ?512 ? _vnni ? rm_fma ?vx ? ?? ?? ?? ann ? uda ? ?? ?? ?? 16c ? ma ? p16_va ? ?? ?? ?? ?? ?? atmul_int8 ? etal ? ?? ?? ?? iscv_v ? pc ? ?? ?? e3 ? se3 ? ?? ?? s ? ve ? ycl ? ?? ?? sx ? ulkan ? ??
a ? blas ? c ? f ? gpublas ? kompute ? llamafile ? m ? neon ? r ? s ? v ? wasm_simd ? get_sve_cnt ?has_ ? ?? u_ ? y ? ?? ??_back ? ?? ??_per_ms ? a ?lamp ?o ?p ? ross_entropy_loss ? ycles ? з ?_inplace ? ?? ?_inplace ? inf ? zero ? е_mask_ ? й ?_inplace ?! ag ? v ?! Н ?? inplace ?!tensor ?! ??_ ?! i ?!up ?! ?? ?? ??_inplace ?! ement_size ?!u ?! ?? ??_inplace ?" l ?!xp ?" ?? ?? ??_set_prec ?" back ?"ext ?" ?? ? pen ?"rmat_name ?" ?? ?~_row ?" ?? ??_ref ?# ?_row ?# ?? ?_row ?# bf16 ?#fp16 ?# 16_to_fp32 ?#32_to_ ?# ?? И lash_attn_ ?"o ?"p ?#ree ?#type_to_ggml_type ?# ? ?? ?? ?? ??_n ?$ ?? ?_n ?$ alloc_graph ?$free ?$get_buffer_size ?$new ?$reserve ?$ ?? ?? ??_inplace ?% inplace ?%quick ?% ??_ ?% ??. ?. 4_q8_0 ?%8_q8_0 ?% ??. 4x ?%8x8_q8_0 ?% ??- ??- 4_q8_0 ?&8_q8_0 ?& ??. 4x ?&8x8_q8_0 ?& m_q4_0_ ?%v_q4_0_ ?& ?? И_f32 ?& ?? ?? 1d ?&nd ?& ? 32_ ?&irst_tensor ?& ?? ?? 1d ?'nd ?' ?? ?? ?? buffer ?'size ?' ax_tensor_size ?'em_ ?' ? ?? ?? ame ?'ext_tensor ?'o_alloc ?' ? ?? ??_back ?( el_pos ?(ows ?( ?? Ж data ?&f ?'i32_ ?'m ?'n ?'r ?(tensor ?(unary_op ?( lu ?%m ?&t_ ?( ?? ?? ?? ??_with_ctx ?) ?? lear ?)ompute ?)py ?) ?? ?? mp_dot ?)p ?) н ?? ?? ?? ?? ??s ?* _nodes ?*ode ?* ?? ??_custom ?* ?? ?? lan ?*rint ?* ?? ?? ?? add_node ?)c ?)du ?)export ?)get_tensor ?)import ?*n ?*overhead ?*p ?*reset ?*size ?*view ?* ?? ??_inplace ?+ aph_ ?*oup_norm ?+ Ћ allocr_ ?$e ?)r ?+uid_matches ?+ ?? ?? igmoid ?,wish ?, ?? ?? ?? free ?,new ?,reset ?, ?? et_ ?,ize ?, rds ?,sh_s ?, ?? ??_back ?- ?? Ѝ it ?-ternal_get_type_traits ?- ?? ?? ?? Н 0 ?-1 ?-2 ?- Й_ ?- ?? ? ?? ? ?? ?? ?? З
3d ?-contiguous ?-empty ?-matrix ?.numa ?.permuted ?.quantized ?.scalar ?.transposed ?.vector ?. m2col ?-n ?-s_ ?. ?? ?~ ?? ?{ place ?/ternal ?/ ?? callback_default ?/in ?/set ?/ ?_ ?/ eaky_relu ?/og ?/ ? ?? f32 ?0inplace_f32 ?0 ?? ?? ?_f32 ?0 f32 ?0inplace ?0 ??_ ?0 ?? ?? ??_f32 ?0 f32 ?0inplace ?0 н_ ?0 з ? ??_f32 ?1 f32 ?1inplace ?1 ??_ ?1 1 ?02 ?13 ?1 ? Я f32 ?1inplace_f32 ?1 binary_ ?0custom ?1unary_ ?1 ?? ??K ??5 end ?2start ?2 an ?2tallib_ ?2 ? ?? ?? id ?2set_prec ?2 ??_ ?2 inplace ?2mat ?2 ??_ ?2 ap_ ?1e ?2ul ?3 ?? В ??_pad ?3 ?? ??_inplace ?3 ?? ?? ?? ??_custom ?3 ?? ?? ?? ?? ?? 1d ?42d ?43d ?44d ?4 ?_ ?4 f32 ?3graph ?3i32 ?3tensor ?4 g ?3lements ?3w_ ?4 ?? ??_inplace ?4 ?? ?? _dims ?3bytes ?3e ?4orm ?4rows ?5uma_init ?5 ?? ?? ?? desc ?5name ?5symbol ?5 ?? ?? ?? ??_g ?5 ?? default_params ?5init ?5resume ?5step_adamw ?6 ??_ ?6 _ ?5t ?6 ?? p ?6ut_prod ?6 ?? Щ ?? ?? ??_back ?6 1d ?62d ?6 ?? ??s ?7 ad ?6ermute ?6ool_ ?7rint_object ?7 ? ? ? ?? chunk ?7free ?7init ?7requires_imatrix ?7 ?? ??_inplace ?8 ?? ??_back ?8 ?? ?? ?? ?? 1d ?82d ?83d ?84d ?8 ??_ ?8 lu ?8peat ?8shape ?8 ?? ?? back ?9inplace ?9 ??_ ?9 ?? ?? ??_inplace ?9 ?? ??_inplace ?9 ?? ?? back ?9custom ?9ext ?9inplace ?9yarn_corr_dims ?9 ??_ ?9 ? pe ?:w_size ?: ?? e ?8ms_norm ?9o ?:wkv_wkv ?: ?? Ѐ_inplace ?: Ї ??_inplace ?; ?? ??_inplace ?; А ? 1d ?;nd ?; ??_ ?; ?? ?? 1d ?;nd ?; ??_ ?; ?? ?? lace ?;ut ?; 32 ?;np ?; ? ?? ?? ame ? ?? ?? back ?>ext ?>inplace ?> к_ ?> ?? ? ??_inplace ?> ?_inplace ?>t ?> ?? ?? conv ??scan ?? ?~ ?? ??_inplace ?? atus_to_string ??ep ?? ? ??_inplace ?? ?? ??_rows ?@ b ??m ?@ cale ?:et ?=gn ?=i ?>oft_max ?>qr ??sm_ ??t ??u ?@ ??\ ?? ?? alloc ?@new ?@ ?? ??_inplace ?@ ble_f32_f16 ?@llocr_ ?@nh ?@ ?? ?? ?? ?? ?? ?? default ?Ainit ?Amatch ?A ?? rams_ ?Ause ?A ?? free ?Anew ?Apa ?Aresume ?B ?? ?? Ќ init ?Bms ?Bus ?B ?? _ ?Bstep_embedding ?B ?? ? ? ?? ??f ?B name ?Bsize ?B a ?Aensor_overhead ?Ahreadpool_ ?Bime ?Bop_k ?Branspose ?Bype_ ?C ?? ?? inplace ?Cop_name ?C ??_ ?C ?? ary ?Cravel_index ?D ?? ??_ext ?D ?? n ?Dpscale ?Dsed_mem ?D ??( ??$ ??$ m_q8_K ?Ds_q8_K ?D ??$ ??$ ??$ s_q8_K ?Exs_q8_K ?E s_q8_K ?Dx ?E ??$ ??$ s_q8_K ?Exxs_q8_K ?E ??$ ??$ nl_q8_0 ?Exs_q8_K ?E 1_ ?D2_ ?E3_ ?E4_ ?E Б$ ?$ ??# ??# ?$ 0_q8_0 ?F1_q8_1 ?FK_q8_K ?F ??# ??$ ??$ 0_q8_0 ?F1_q8_1 ?FK_q8_K ?F ??$ ??$ 2_K_q8_K ?F3_K_q8_K ?F4_ ?F5_ ?F6_K_q8_K ?F8_0_q8_0 ?F ??$ ??$ 1_0_q8_K ?G2_0_q8_K ?G iq ?Eq ?Gtq ?G ?? ?? ?? ? ?? 1d ?G2d ?G3d ?G4d ?Gtensor ?H alidate_row_data ?Dec_dot_ ?Giew_ ?H ?? ?? part ?Hunpart ?H a ?b ?c ? d ?!e ?"f ?#g ?,ha ?,i ?/l ?/m ?3n ?5o ?6p ?7quantize_ ?7r ?:s ?@t ?Cu ?Dv ?Hwin_ ?H ?? ?? ?? key ?Itensor ?I ?? ind_ ?Iree ?I ?? ?? ?? ?? ?? data ?Jn ?Jstr ?Jtype ?J lignment ?Jrr_ ?J ?? ??_offset ?J ?? ?? ey ?Jv_type ?J ?? ?? data ?Ksize ?K ?? ?? kv ?Ktensors ?K ?? ?? ?? name ?Koffset ?Ktype ?K ?? ?? ?? ?? 32 ?K64 ?K ?? ?? ?? ?? 16 ?L32 ?L64 ?L8 ?L ?? ?? ?? ?? ?? 16 ?L32 ?L64 ?L8 ?L bool ?Kdata ?Kf ?Li ?Lstr ?Lu ?L ?? al_ ?Lersion ?M a ?Jdata ?Jk ?Jmeta_ ?Kn_ ?Ktensor_ ?Kv ?M ? и empty ?Mfrom_file ?M ?? ?? ?? data ?Mstr ?M ?? ?? ?? data ?Ntype ?N Љ ?? ?? 32 ?N64 ?N ?? ?? ?? ?? 16 ?N32 ?N64 ?N8 ?N ?? ?? ?? ?? ?? 16 ?O32 ?O64 ?O8 ?O bool ?Nf ?Ni ?Nstr ?Ou ?O arr_ ?Nkv ?Ntensor_ ?Nval_ ?O ?? ?? add_tensor ?If ?Iget_ ?Minit_ ?Mremove_key ?Mset_ ?Otype_name ?Owrite_to_file ?O ml_ ?Huf_ ?O ??% ??$ free_impl ?Pinit_impl ?P ??& ??& free_impl ?Qinit_impl ?Q 2xs_ ?P3xs_ ?Q ??' ??& m ?Qs ?Q ??( ??% ??% s ?Qxs ?Q s ?Qx ?Q ??& ??& s ?Qxxs ?Q ??' ??( nl ?Rxs ?R 1_ ?Q2_ ?Q3_ ?Q4_ ?R ??- ?? ?? ??- ??- 4 ?R8 ?R ??- 4x ?R8x8 ?R ??!_ ?R ??! ?? 0 ?R1 ?RK ?S ?" ?" ?? 0 ?S1 ?SK ?S ?? ??- ??- 4 ?S8 ?S ??"_4x ?S 2_K ?R3_K ?R4_ ?S5_ ?S6_K ?S8_0 ?S ??( ??(_ref ?T ??& ??&_ref ?T ??& ??&_ref ?T s ?Txxs ?T ??( Г(_ref ?T ?( ??(_ref ?T nl ?Txs ?T 2_s ?T3_ ?T4_ ?T ?? ??_ref ?U ?? ??_ref ?U ?? ??_ref ?U ?? ??_ref ?U ?? ?? _ref ?U 0 ?U1 ?UK ?U ?? ??_ref ?U ?? ??_ref ?V ?? ?? _ref ?V 0 ?V1 ?VK ?V ? ?? _ref ?V ?? ??_ref ?V ?? ??_ref ?V ??# ??#_ref ?V 0 ?V1 ?VK ?W 2_K ?U3_K ?U4_ ?U5_ ?V6_K ?V8_ ?W ??" ??"_ref ?W ??" ??"_ref ?W 1_0 ?W2_0 ?W iq ?Tq ?Wtq ?W ??" ??" 1_0 ?X2_0 ?X iq ?Rmat_q8_0 ?Rq ?Srow_ ?Wtq ?X OBJC_ dequantize_row_ ?gg ?Piq ?Qquantize_ ?X ?y??00 @????? PPp00? ? ?@@0 00 0 0P ????00@@???
0?@@? 0P``???? ??????????p?p`@???? ? ???@?@?@?@?@?@??p??????????????????@???p?p??P??0??? ? 00 ?``??????????????????`?`? ? @?@?@ ?????????0P???? ????@??0 ??
?????p??p????????????????`? 0???`?`??p? `@?0p@?@@???`? ??? ???? ????????p?` ?p? ??pp? PPpp?pppppppppppp?p ????????????????????? @?0?????????? ?#????? ??"?? p??0? `00 ` ? 0 ????? @@@0000?? @@@ ??????`??``pp???? `pp??? ?p???aP?`?
@`?????? ?@?P@P0@ ` @??0? P??? ?p@??? ??
??????$?????? ?? ??:?*??*?-? ?,?'?
???? ?$? ?*`? ?``??
????????? ???????????????????????? 0??4?E?0???,?4 ?4?5??)??????@ ?2??0? ?p?
?У pp`?? P ??
????? `??"?@P0??P??0????? ????, ?A ?] ?A `? ?A ?? ?A @? ?A ? ?A ? ?A ? ?A `? B ?? B p? ,B @? V pf ?V `g ?V ? pg 'W ?g ?W ?g 9X h ?X @h cY i dZ ?i ^[ P? |[ p? ?[ ?? ?[ ?? ?[ б \ `? \ x? 6\ н F\ P? S\ P? a\ ?? m\ ?? x\ ? ?\ ? ?\ ? ?\ ?[ ?\ ?[ ?\ ?_ ?\ ?a ?\ ?e ] ?u >] ?} Y] ? t] ?? ?] ?? ?] 9^ ?? L^ ?? _^ ?? r^ ?? ?^ ? ?^ ? ?^ (? ?^ 8? ?^ L? ?^ `? ?^ t? _ ?? _ ?? 3_ ?? G_ ?? [_ ?? o_ ?? ?_ @? ?_ P? ?_ `? ?_ ?? ?_ ?
?_ ?
?_
?_ ?
`
%` ?
N` ?
p` H
?` ?
?`
?` ? ?
?` ?
?` @
?a p
b ?C
4b D
Ub ?F
db ?F
vb ?F
?b ?F
?b ?F
c G
(c `G
Fc pG
kc P? tc ?: ?c ?: ?c ?: ?c ?: ?c ?: ?c ?: #d ?: gd ?: {d ; ?d ; ?d 8; e @; he ?; ?e ?; ?e P< ?e ?< ?e ?< f ?< PF
(F
@ P? V ?? l ?? ? 0? ? p? ? ?? ? 0? ? ?? ? @?
0? " ?? 7 Ў L @? a p
v ? ? @? ? ?A ? @v ? p? ? P? ? @?
?? ?< , ? 6 ?? H ? R ? d P? n ?? y ? ? @? ? 0? ? @ ? ? ? @? ? ?O ? P ? p? , ?? G ?? g ? ? ?S ? PT ? ?? ? 0? ?? p? C ?? g ? ? ?? ? ? ? ?? ? ? p? P? = ?? Z `? ?? ? ? ? ? ? ?? ? ?? ? 6 `? T ? t @? ? ?? ? `a ? p
? ? @ " 0 8
] 0 } ? ? ?? ? ? ? ?? p ? 5 M ?? k ? ? ?? ? ?? ? ?? ? ?? ? p? ?? $ ?? D @? b ?? y ? ? ?? ? p? ? ?? ? ??
0?
?? /
?? K
?? q
?? ?
?? ?
?? ?
? ?
?V ?
a ?? ; ? [ P? y `? ? ? 0 ? ? ? 0T ? ?
?^
@ \
` ?
? d
? ?
? p?
?
?? :
?? M
?? f
? ?
0 ?
? ?
0? ?
? ?
@? @?
$ ? ; `? R ?H r ? ? ?Q ? R ? ?Q ? ?Q ?Q 4 ?S [ ?J } ?J ? P ? ? p ? ?Q pR @ ?H ` P? | ? ? ?? ? pU ? ?? ? p? ?? p? = ?? Y ?? r ? ? ?T ? ?? ? 0B ? I ? ?2 ? ?* + 0g F ?g \ PP m `? x @? ? ?? ? ?? ? ? ? ?? ? ?? ? ? ? ? ? p? ? ? `? 0? ) ?? A ? Y ?? t ?? ~ ? ? P? ? ?$ ? ?# ? ?" ? # ? # @# " # < 0# V ?" m ?# ? @$ ? ?# ? ?# ? P# ? ?# ? `$ ? $
P$ % ?$ ? ?# S `# f ?# | 0$ ? p$ ? ?$ ? p# ? $ ? ?$ ? $ ?# @? # ?" < 0# Z ?F g ?F { ?? ? p? ? ?? ? `? ? ?? ? ? ? М ?? Ў ?x 1 ?K D ? N ? ` ? j ? | P? ? ? ? ? ? ?F ? ?? ? p? ? @
?? ?D 7 ?C R ?? e ?@ | U ? PL ? p? ? ?? ? ?? ? `? ? @? `? ' `? ? ? J в ] ? n г ? @? ? ?? ? 0? ? 0{ ? 0? P? , `? ? ?? P 0? a `] x { ? ?~ ? @\ ? \ ? 0\ ? p? ? ?] \ p
& ?? 5 ?? I 0? Z PK m `o ? ?n ? ?? ? 0] ? ?. ? p| ? l ? ?^
^ $ ?l 7 Po K ?n \ @o n i ? ?h ? q ? { ? `l ? ?n ? ?k ? ?? к ?E 3 ? E ? U `* i ?' | @* ? @) ? ?? ? ? ? 0Q ? ?F ? L ? ?L
?M 0N 6 ?N L ?O [ ?K k pG y pO ? ?J ? ?K ? ?L ? ?K ? ? ? `? ? ? ?? ?= - ?$ ; ` P ? m ? ? ? ? ? ? ? ? ? ? 0 - ? ? ? U ?! o ` ? ` ? ? ? `? ? O? ? ?G
? ? ? ?? ! p? ! ? +! ?? B! L O! ?H \! PI m! ? w! ? ?! @H ?! ?n ?! ?k ?! ?i ?! e ?! ?] ?! ?c " d " `d )" ?d =" ? H" p? [" `H g" @G w" K ?" ?J ?" ?J ?" @? ?" ?? ?" ?? ?" `? ?" @? ?" $ # ? # ?? '# ?? 5# ?? C# ?? Q# @? d# ?G w# ?G ?# `? ?# ?? ?# `? ?# @? ?# ? ?# ? $ ? $ @? %$ ?? 3$ ? D$ ?? U$ @? f$ ?? w$ ? ?$ ? ?$ p? ?$ ? ?$ ? ?$ ?? ?$ ? ?$ ?? % ?? % `? 1% ?& K% ?I Z% ? i% P? u% ?? ?% ?? ?% ?? ?% ?? ?% ? ?% ? ?% po ?% P? ?% ?? & ?e & ? && `? 7& ?? I& ?? Y& ?? h& ?? w& \ ?& ?? ?& ?? ?& ?[ ?& ?x ?& ? ?& ? ?& ? ?& б ' ? ' ? '' д :' ? D' `? V' P? e' 0? y' p? ?' ? ?' ? ?' ?? ?' ? ?' ?? ?' ?? ?' p? ( ? ( @? .( ? 9( ? L( ? V( ? h( ?? r( ? ?( ?< ?( ?? ?( ? ?( ? ?( ? ?( ?L ?( ?o ) ?{ ) ?z :) z W) P{ u) ?p ?) ?p ?) ?E ?) F ?) PF ?) ?? ?) ?? ?) ?? * pJ * 0I %* 0J 6* Ч B* Ш V* K j* `z ~* P? ?* ?? ?* ?[ ?* ?/
?* `) ?* ?& ?* ? + ? *+ ? E+ p# ^+ ` y+ ?- ?+ ?1 ?+ ? ?+ ?
?+ ? ?+ ??
, ?
%, ?? =, ? U, ? m, ? ?, ? ?, ?, ? ?, ?? ?, ? ?, ? ?, ?? - ?? - ? (- ? 9- 0 J- ? Y- ?? k- P? v- ?? ?- ? ?- ? ?- ?? ?- ?? ?- ?? ?- ?? ?- ? . P? . 0" +. ?! ?. ?? N. ?? b. `? x. ?? ?. ?? ?. `? ?. @? ?. ?? ?. ?? ?. ?? / 0? / ?? &/ ?? 7/ ?? I/ P? [/ p? m/ ? / p? ?/ ?? ?/ ?? ?/ P? ?/ ?? ?/ ?/ ?/ 0 0 !0 ? 70 ? J0 p \0 ? n0 @? ?0 `? ?0 ? ?0 ? ?0 ? ?0 0? ?0 P? ?0 ? ?0 ? 1 ?? 1 21 ?E C1 P5 T1 `? e1 `? v1 ?? ?1 @? ?1 `
?1 ?` ?1 ?F ?1 ? ?1 ? ?1 ?? ?1 ?
2 ?q 2 ?? /2 ? >2 n M2 r `2 u s2 ?w ?2 @| ?2 ? ?2 p? ?2 p? ?2 0G ?2 p ?2 ?? ?2 ?i 3 ?m 3 ?/
)3 p/
A3 ? U3 ? m3 У ?3 ? ?3 ?
?3 ?
?3 ?
?3 p
?3 ?? 4 ? #4 ? 64 0? M4 ?w `4 ?r w4 p{ ?4 ?w ?4 ? ?4 @? ?4 ?? ?4 ?{ ?4 0? 5 ?? 5 G 25 ?* I5 ` \5 `] s5 `? ?5 @? ?5 @? ?5 ?? ?5 ? ?5 ?? ?5 `? 6 ? 6 p? 16 ? I6 ?? Y6 ? i6 |6 ?6 ?6 ?6 ?6 7 67 M7 d7 x7 ?7 ?7 ?7 ?7 ?7 8 (8 I8 ?8 ?8 ?8 ?8 29 d9 ?9 ?9 ?9 ?9 : 3: N: ~: ?: ?: ?: ?: ?: ; *; H; r; ?; ?; ?; ?; ?; ?; ?; ?; < < ,< ?< P< f< y< ?< ?< ?< ?< ?< ?< ?< = = = *= K= ^= e= m= @ ?= ?= ?= ?= ?= ?= ?= ?= ?= ?= ?= > > !> (> /> 6> <> C> J> R> Z> b> h> o> u> ~> ?> ?> ?> ?> ?> ?> ?> ?> ? ? (? 8? F? T? d? j? r? ?? ?? ?? ?? ?? ?? ?? @ #@ 9@ G@ ^@ s@ |@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ ?@ A
A A #A .A :A EA OA \A kA vA v w ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? v w ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
_OBJC_CLASS_$_GGMLMetalClass _OBJC_METACLASS_$_GGMLMetalClass _dequantize_row_iq1_m _dequantize_row_iq1_s _dequantize_row_iq2_s _dequantize_row_iq2_xs _dequantize_row_iq2_xxs _dequantize_row_iq3_s _dequantize_row_iq3_xxs _dequantize_row_iq4_nl _dequantize_row_iq4_xs _dequantize_row_q2_K _dequantize_row_q3_K _dequantize_row_q4_0 _dequantize_row_q4_1 _dequantize_row_q4_K _dequantize_row_q5_0 _dequantize_row_q5_1 _dequantize_row_q5_K _dequantize_row_q6_K _dequantize_row_q8_0 _dequantize_row_q8_K _dequantize_row_tq1_0 _dequantize_row_tq2_0 _ggml_abort _ggml_abs _ggml_abs_inplace _ggml_acc _ggml_acc_inplace _ggml_add _ggml_add1 _ggml_add1_inplace _ggml_add_cast _ggml_add_inplace _ggml_add_rel_pos _ggml_add_rel_pos_inplace _ggml_arange _ggml_are_same_shape _ggml_are_same_stride _ggml_argmax _ggml_argsort _ggml_backend_alloc_buffer _ggml_backend_alloc_ctx_tensors _ggml_backend_alloc_ctx_tensors_from_buft _ggml_backend_blas_init _ggml_backend_blas_set_n_threads _ggml_backend_buffer_clear _ggml_backend_buffer_copy_tensor _ggml_backend_buffer_free _ggml_backend_buffer_get_alignment _ggml_backend_buffer_get_alloc_size _ggml_backend_buffer_get_base _ggml_backend_buffer_get_max_size _ggml_backend_buffer_get_size _ggml_backend_buffer_get_type _ggml_backend_buffer_get_usage _ggml_backend_buffer_init _ggml_backend_buffer_init_tensor _ggml_backend_buffer_is_host _ggml_backend_buffer_is_multi_buffer _ggml_backend_buffer_name _ggml_backend_buffer_reset _ggml_backend_buffer_set_usage _ggml_backend_buft_alloc_buffer _ggml_backend_buft_get_alignment _ggml_backend_buft_get_alloc_size _ggml_backend_buft_get_device _ggml_backend_buft_get_max_size _ggml_backend_buft_is_host _ggml_backend_buft_name _ggml_backend_compare_graph_backend _ggml_backend_cpu_buffer_from_ptr _ggml_backend_cpu_buffer_type _ggml_backend_cpu_init _ggml_backend_cpu_reg _ggml_backend_cpu_set_abort_callback _ggml_backend_cpu_set_n_threads _ggml_backend_cpu_set_threadpool _ggml_backend_dev_backend_reg _ggml_backend_dev_buffer_from_host_ptr _ggml_backend_dev_buffer_type _ggml_backend_dev_by_name _ggml_backend_dev_by_type _ggml_backend_dev_count _ggml_backend_dev_description _ggml_backend_dev_get _ggml_backend_dev_get_props _ggml_backend_dev_host_buffer_type _ggml_backend_dev_init _ggml_backend_dev_memory _ggml_backend_dev_name _ggml_backend_dev_offload_op _ggml_backend_dev_supports_buft _ggml_backend_dev_supports_op _ggml_backend_dev_type _ggml_backend_device_register _ggml_backend_event_free _ggml_backend_event_new _ggml_backend_event_record _ggml_backend_event_synchronize _ggml_backend_event_wait _ggml_backend_free _ggml_backend_get_alignment _ggml_backend_get_default_buffer_type _ggml_backend_get_device _ggml_backend_get_max_size _ggml_backend_graph_compute _ggml_backend_graph_compute_async _ggml_backend_graph_copy _ggml_backend_graph_copy_free _ggml_backend_graph_plan_compute _ggml_backend_graph_plan_create _ggml_backend_graph_plan_free _ggml_backend_guid _ggml_backend_init_best _ggml_backend_init_by_name _ggml_backend_init_by_type _ggml_backend_is_blas _ggml_backend_is_cpu _ggml_backend_is_metal _ggml_backend_metal_buffer_from_ptr _ggml_backend_metal_buffer_type _ggml_backend_metal_capture_next_compute _ggml_backend_metal_init _ggml_backend_metal_set_abort_callback _ggml_backend_metal_supports_family _ggml_backend_multi_buffer_alloc_buffer _ggml_backend_multi_buffer_set_usage _ggml_backend_name _ggml_backend_offload_op _ggml_backend_reg_by_name _ggml_backend_reg_count _ggml_backend_reg_dev_count _ggml_backend_reg_dev_get _ggml_backend_reg_get _ggml_backend_reg_get_proc_address _ggml_backend_reg_metal_init _ggml_backend_reg_name _ggml_backend_register _ggml_backend_sched_alloc_graph _ggml_backend_sched_free _ggml_backend_sched_get_backend _ggml_backend_sched_get_buffer_size _ggml_backend_sched_get_n_backends _ggml_backend_sched_get_n_copies _ggml_backend_sched_get_n_splits _ggml_backend_sched_get_tensor_backend _ggml_backend_sched_graph_compute _ggml_backend_sched_graph_compute_async _ggml_backend_sched_new _ggml_backend_sched_reserve _ggml_backend_sched_reset _ggml_backend_sched_set_eval_callback _ggml_backend_sched_set_tensor_backend _ggml_backend_sched_synchronize _ggml_backend_supports_buft _ggml_backend_supports_op _ggml_backend_synchronize _ggml_backend_tensor_alloc _ggml_backend_tensor_copy _ggml_backend_tensor_copy_async _ggml_backend_tensor_get _ggml_backend_tensor_get_async _ggml_backend_tensor_memset _ggml_backend_tensor_set _ggml_backend_tensor_set_async _ggml_backend_view_init _ggml_bf16_to_fp32 _ggml_bf16_to_fp32_row _ggml_blck_size _ggml_build_backward_expand _ggml_build_backward_gradient_checkpointing _ggml_build_forward_expand _ggml_build_opt_adamw _ggml_can_repeat _ggml_cast _ggml_clamp _ggml_concat _ggml_cont _ggml_cont_1d _ggml_cont_2d _ggml_cont_3d _ggml_cont_4d _ggml_conv_1d _ggml_conv_1d_ph _ggml_conv_2d _ggml_conv_2d_s1_ph _ggml_conv_2d_sk_p0 _ggml_conv_depthwise_2d _ggml_conv_transpose_1d _ggml_conv_transpose_2d_p0 _ggml_cos _ggml_cos_inplace _ggml_count_equal _ggml_cpu_get_sve_cnt _ggml_cpu_has_arm_fma _ggml_cpu_has_avx _ggml_cpu_has_avx2 _ggml_cpu_has_avx512 _ggml_cpu_has_avx512_bf16 _ggml_cpu_has_avx512_vbmi _ggml_cpu_has_avx512_vnni _ggml_cpu_has_avx_vnni _ggml_cpu_has_blas _ggml_cpu_has_cann _ggml_cpu_has_cuda _ggml_cpu_has_f16c _ggml_cpu_has_fma _ggml_cpu_has_fp16_va _ggml_cpu_has_gpublas _ggml_cpu_has_kompute _ggml_cpu_has_llamafile _ggml_cpu_has_matmul_int8 _ggml_cpu_has_metal _ggml_cpu_has_neon _ggml_cpu_has_riscv_v _ggml_cpu_has_rpc _ggml_cpu_has_sse3 _ggml_cpu_has_ssse3 _ggml_cpu_has_sve _ggml_cpu_has_sycl _ggml_cpu_has_vsx _ggml_cpu_has_vulkan _ggml_cpu_has_wasm_simd _ggml_cpy _ggml_cross_entropy_loss _ggml_cross_entropy_loss_back _ggml_cycles _ggml_cycles_per_ms _ggml_diag _ggml_diag_mask_inf _ggml_diag_mask_inf_inplace _ggml_diag_mask_zero _ggml_diag_mask_zero_inplace _ggml_div _ggml_div_inplace _ggml_dup _ggml_dup_inplace _ggml_dup_tensor _ggml_element_size _ggml_elu _ggml_elu_inplace _ggml_exp _ggml_exp_inplace _ggml_flash_attn_back _ggml_flash_attn_ext _ggml_flash_attn_ext_set_prec _ggml_fopen _ggml_format_name _ggml_fp16_to_fp32 _ggml_fp16_to_fp32_row _ggml_fp32_to_bf16 _ggml_fp32_to_bf16_row _ggml_fp32_to_bf16_row_ref _ggml_fp32_to_fp16 _ggml_fp32_to_fp16_row _ggml_free _ggml_ftype_to_ggml_type _ggml_gallocr_alloc_graph _ggml_gallocr_free _ggml_gallocr_get_buffer_size _ggml_gallocr_new _ggml_gallocr_new_n _ggml_gallocr_reserve _ggml_gallocr_reserve_n _ggml_gelu _ggml_gelu_inplace _ggml_gelu_quick _ggml_gelu_quick_inplace _ggml_gemm_q4_0_4x4_q8_0 _ggml_gemm_q4_0_4x8_q8_0 _ggml_gemm_q4_0_8x8_q8_0 _ggml_gemv_q4_0_4x4_q8_0 _ggml_gemv_q4_0_4x8_q8_0 _ggml_gemv_q4_0_8x8_q8_0 _ggml_get_data _ggml_get_data_f32 _ggml_get_f32_1d _ggml_get_f32_nd _ggml_get_first_tensor _ggml_get_i32_1d _ggml_get_i32_nd _ggml_get_max_tensor_size _ggml_get_mem_buffer _ggml_get_mem_size _ggml_get_name _ggml_get_next_tensor _ggml_get_no_alloc _ggml_get_rel_pos _ggml_get_rows _ggml_get_rows_back _ggml_get_tensor _ggml_get_unary_op _ggml_graph_add_node _ggml_graph_clear _ggml_graph_compute _ggml_graph_compute_with_ctx _ggml_graph_cpy _ggml_graph_dump_dot _ggml_graph_dup _ggml_graph_export _ggml_graph_get_tensor _ggml_graph_import _ggml_graph_n_nodes _ggml_graph_node _ggml_graph_nodes _ggml_graph_overhead _ggml_graph_overhead_custom _ggml_graph_plan _ggml_graph_print _ggml_graph_reset _ggml_graph_size _ggml_graph_view _ggml_group_norm _ggml_group_norm_inplace _ggml_guid_matches _ggml_hardsigmoid _ggml_hardswish _ggml_hash_set_free _ggml_hash_set_new _ggml_hash_set_reset _ggml_hash_size _ggml_im2col _ggml_im2col_back _ggml_init _ggml_internal_get_type_traits _ggml_is_3d _ggml_is_contiguous _ggml_is_contiguous_0 _ggml_is_contiguous_1 _ggml_is_contiguous_2 _ggml_is_empty _ggml_is_matrix _ggml_is_numa _ggml_is_permuted _ggml_is_quantized _ggml_is_scalar _ggml_is_transposed _ggml_is_vector _ggml_leaky_relu _ggml_log _ggml_log_callback_default _ggml_log_inplace _ggml_log_internal _ggml_log_set _ggml_map_binary_f32 _ggml_map_binary_inplace_f32 _ggml_map_custom1 _ggml_map_custom1_f32 _ggml_map_custom1_inplace _ggml_map_custom1_inplace_f32 _ggml_map_custom2 _ggml_map_custom2_f32 _ggml_map_custom2_inplace _ggml_map_custom2_inplace_f32 _ggml_map_custom3 _ggml_map_custom3_f32 _ggml_map_custom3_inplace _ggml_map_custom3_inplace_f32 _ggml_map_unary_f32 _ggml_map_unary_inplace_f32 _ggml_mean _ggml_metallib_end _ggml_metallib_start _ggml_mul _ggml_mul_inplace _ggml_mul_mat _ggml_mul_mat_id _ggml_mul_mat_set_prec _ggml_n_dims _ggml_nbytes _ggml_nbytes_pad _ggml_neg _ggml_neg_inplace _ggml_nelements _ggml_new_f32 _ggml_new_graph _ggml_new_graph_custom _ggml_new_i32 _ggml_new_tensor _ggml_new_tensor_1d _ggml_new_tensor_2d _ggml_new_tensor_3d _ggml_new_tensor_4d _ggml_norm _ggml_norm_inplace _ggml_nrows _ggml_numa_init _ggml_op_desc _ggml_op_name _ggml_op_symbol _ggml_opt _ggml_opt_default_params _ggml_opt_init _ggml_opt_resume _ggml_opt_resume_g _ggml_opt_step_adamw _ggml_out_prod _ggml_pad _ggml_permute _ggml_pool_1d _ggml_pool_2d _ggml_pool_2d_back _ggml_print_object _ggml_print_objects _ggml_quantize_chunk _ggml_quantize_free _ggml_quantize_init _ggml_quantize_requires_imatrix _ggml_relu _ggml_relu_inplace _ggml_repeat _ggml_repeat_back _ggml_reshape _ggml_reshape_1d _ggml_reshape_2d _ggml_reshape_3d _ggml_reshape_4d _ggml_rms_norm _ggml_rms_norm_back _ggml_rms_norm_inplace _ggml_rope _ggml_rope_back _ggml_rope_custom _ggml_rope_custom_inplace _ggml_rope_ext _ggml_rope_ext_inplace _ggml_rope_inplace _ggml_rope_yarn_corr_dims _ggml_row_size _ggml_rwkv_wkv _ggml_scale _ggml_scale_inplace _ggml_set _ggml_set_1d _ggml_set_1d_inplace _ggml_set_2d _ggml_set_2d_inplace _ggml_set_f32 _ggml_set_f32_1d _ggml_set_f32_nd _ggml_set_i32 _ggml_set_i32_1d _ggml_set_i32_nd _ggml_set_inplace _ggml_set_input _ggml_set_loss _ggml_set_name _ggml_set_no_alloc _ggml_set_output _ggml_set_param _ggml_set_scratch _ggml_set_zero _ggml_sgn _ggml_sgn_inplace _ggml_sigmoid _ggml_sigmoid_inplace _ggml_silu _ggml_silu_back _ggml_silu_inplace _ggml_sin _ggml_sin_inplace _ggml_soft_max _ggml_soft_max_back _ggml_soft_max_back_inplace _ggml_soft_max_ext _ggml_soft_max_inplace _ggml_sqr _ggml_sqr_inplace _ggml_sqrt _ggml_sqrt_inplace _ggml_ssm_conv _ggml_ssm_scan _ggml_status_to_string _ggml_step _ggml_step_inplace _ggml_sub _ggml_sub_inplace _ggml_sum _ggml_sum_rows _ggml_table_f32_f16 _ggml_tallocr_alloc _ggml_tallocr_new _ggml_tanh _ggml_tanh_inplace _ggml_tensor_overhead _ggml_threadpool_free _ggml_threadpool_new _ggml_threadpool_params_default _ggml_threadpool_params_init _ggml_threadpool_params_match _ggml_threadpool_pause _ggml_threadpool_resume _ggml_time_init _ggml_time_ms _ggml_time_us _ggml_timestep_embedding _ggml_top_k _ggml_transpose _ggml_type_name _ggml_type_size _ggml_type_sizef _ggml_unary _ggml_unary_inplace _ggml_unary_op_name _ggml_unravel_index _ggml_upscale _ggml_upscale_ext _ggml_used_mem _ggml_validate_row_data _ggml_vec_dot_iq1_m_q8_K _ggml_vec_dot_iq1_s_q8_K _ggml_vec_dot_iq2_s_q8_K _ggml_vec_dot_iq2_xs_q8_K _ggml_vec_dot_iq2_xxs_q8_K _ggml_vec_dot_iq3_s_q8_K _ggml_vec_dot_iq3_xxs_q8_K _ggml_vec_dot_iq4_nl_q8_0 _ggml_vec_dot_iq4_xs_q8_K _ggml_vec_dot_q2_K_q8_K _ggml_vec_dot_q3_K_q8_K _ggml_vec_dot_q4_0_q8_0 _ggml_vec_dot_q4_1_q8_1 _ggml_vec_dot_q4_K_q8_K _ggml_vec_dot_q5_0_q8_0 _ggml_vec_dot_q5_1_q8_1 _ggml_vec_dot_q5_K_q8_K _ggml_vec_dot_q6_K_q8_K _ggml_vec_dot_q8_0_q8_0 _ggml_vec_dot_tq1_0_q8_K _ggml_vec_dot_tq2_0_q8_K _ggml_view_1d _ggml_view_2d _ggml_view_3d _ggml_view_4d _ggml_view_tensor _ggml_win_part _ggml_win_unpart _gguf_add_tensor _gguf_find_key _gguf_find_tensor _gguf_free _gguf_get_alignment _gguf_get_arr_data _gguf_get_arr_n _gguf_get_arr_str _gguf_get_arr_type _gguf_get_data _gguf_get_data_offset _gguf_get_key _gguf_get_kv_type _gguf_get_meta_data _gguf_get_meta_size _gguf_get_n_kv _gguf_get_n_tensors _gguf_get_tensor_name _gguf_get_tensor_offset _gguf_get_tensor_type _gguf_get_val_bool _gguf_get_val_data _gguf_get_val_f32 _gguf_get_val_f64 _gguf_get_val_i16 _gguf_get_val_i32 _gguf_get_val_i64 _gguf_get_val_i8 _gguf_get_val_str _gguf_get_val_u16 _gguf_get_val_u32 _gguf_get_val_u64 _gguf_get_val_u8 _gguf_get_version _gguf_init_empty _gguf_init_from_file _gguf_remove_key _gguf_set_arr_data _gguf_set_arr_str _gguf_set_kv _gguf_set_tensor_data _gguf_set_tensor_type _gguf_set_val_bool _gguf_set_val_f32 _gguf_set_val_f64 _gguf_set_val_i16 _gguf_set_val_i32 _gguf_set_val_i64 _gguf_set_val_i8 _gguf_set_val_str _gguf_set_val_u16 _gguf_set_val_u32 _gguf_set_val_u64 _gguf_set_val_u8 _gguf_type_name _gguf_write_to_file _iq2xs_free_impl _iq2xs_init_impl _iq3xs_free_impl _iq3xs_init_impl _quantize_iq1_m _quantize_iq1_s _quantize_iq2_s _quantize_iq2_xs _quantize_iq2_xxs _quantize_iq3_s _quantize_iq3_xxs _quantize_iq4_nl _quantize_iq4_xs _quantize_mat_q8_0 _quantize_q2_K _quantize_q3_K _quantize_q4_0 _quantize_q4_0_4x4 _quantize_q4_0_4x8 _quantize_q4_0_8x8 _quantize_q4_1 _quantize_q4_K _quantize_q5_0 _quantize_q5_1 _quantize_q5_K _quantize_q6_K _quantize_q8_0 _quantize_q8_0_4x4 _quantize_q8_0_4x8 _quantize_row_iq2_s _quantize_row_iq2_s_ref _quantize_row_iq3_s _quantize_row_iq3_s_ref _quantize_row_iq3_xxs _quantize_row_iq3_xxs_ref _quantize_row_iq4_nl _quantize_row_iq4_nl_ref _quantize_row_iq4_xs _quantize_row_iq4_xs_ref _quantize_row_q2_K _quantize_row_q2_K_ref _quantize_row_q3_K _quantize_row_q3_K_ref _quantize_row_q4_0 _quantize_row_q4_0_ref _quantize_row_q4_1 _quantize_row_q4_1_ref _quantize_row_q4_K _quantize_row_q4_K_ref _quantize_row_q5_0 _quantize_row_q5_0_ref _quantize_row_q5_1 _quantize_row_q5_1_ref _quantize_row_q5_K _quantize_row_q5_K_ref _quantize_row_q6_K _quantize_row_q6_K_ref _quantize_row_q8_0 _quantize_row_q8_0_ref _quantize_row_q8_1 _quantize_row_q8_1_ref _quantize_row_q8_K _quantize_row_q8_K_ref _quantize_row_tq1_0 _quantize_row_tq1_0_ref _quantize_row_tq2_0 _quantize_row_tq2_0_ref _quantize_tq1_0 _quantize_tq2_0 _MTLCopyAllDevices _MTLCreateSystemDefaultDevice _OBJC_CLASS_$_MTLCaptureDescriptor _OBJC_CLASS_$_MTLCaptureManager _OBJC_CLASS_$_MTLCompileOptions _OBJC_CLASS_$_NSBundle _OBJC_CLASS_$_NSMutableDictionary _OBJC_CLASS_$_NSObject _OBJC_CLASS_$_NSString _OBJC_CLASS_$_NSURL _OBJC_METACLASS_$_NSObject __NSConcreteStackBlock __Unwind_Resume __ZNSt11logic_errorC2EPKc __ZNSt12length_errorD1Ev __ZNSt13exception_ptrD1Ev __ZNSt20bad_array_new_lengthC1Ev __ZNSt20bad_array_new_lengthD1Ev __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEmc __ZNSt3__114__shared_countD2Ev __ZNSt3__115__thread_structC1Ev __ZNSt3__115__thread_structD1Ev __ZNSt3__117__assoc_sub_state13set_exceptionESt13exception_ptr __ZNSt3__117__assoc_sub_state16__on_zero_sharedEv __ZNSt3__117__assoc_sub_state4waitEv __ZNSt3__117__assoc_sub_state9set_valueEv __ZNSt3__118condition_variableD1Ev __ZNSt3__119__thread_local_dataEv __ZNSt3__120__throw_system_errorEiPKc __ZNSt3__15mutexD1Ev __ZNSt3__16futureIvE3getEv __ZNSt3__16futureIvEC1EPNS_17__assoc_sub_stateE __ZNSt3__16futureIvED1Ev __ZNSt3__16thread6detachEv __ZNSt3__16threadD1Ev __ZSt17current_exceptionv __ZSt9terminatev __ZTINSt3__117__assoc_sub_stateE __ZTISt12length_error __ZTISt20bad_array_new_length __ZTVN10__cxxabiv120__si_class_type_infoE __ZTVNSt3__117__assoc_sub_stateE __ZTVSt12length_error __ZdaPv __ZdlPv __Znam __Znwm ___CFConstantStringClassReference ____chkstk_darwin ___bzero ___cxa_allocate_exception ___cxa_atexit ___cxa_begin_catch ___cxa_end_catch ___cxa_free_exception ___cxa_guard_abort ___cxa_guard_acquire ___cxa_guard_release ___cxa_rethrow ___cxa_throw ___error ___gxx_personality_v0 ___sincosf_stret ___stack_chk_fail ___stack_chk_guard ___stderrp ___stdoutp __dispatch_queue_attr_concurrent __objc_empty_cache _abort _calloc _cblas_sgemm$NEWLAPACK$ILP64 _clock _clock_gettime _cosf _dispatch_apply _dispatch_queue_create _dispatch_release _exp2f _expf _expm1f _fclose _fflush _fopen$DARWIN_EXTSN _fprintf _fputc _fputs _fread _free _fseek _ftell _fwrite _getenv _log1pf _log2 _log2f _logf _lroundf _mach_task_self_ _malloc _memcmp _memcpy _memset _memset_pattern16 _objc_alloc _objc_autoreleasePoolPop _objc_autoreleasePoolPush _objc_enumerationMutation _objc_msgSend _objc_opt_class _objc_opt_new _objc_release _posix_memalign _powf _printf _pthread_cond_broadcast _pthread_cond_destroy _pthread_cond_init _pthread_cond_wait _pthread_create _pthread_join _pthread_mutex_destroy _pthread_mutex_init _pthread_mutex_lock _pthread_mutex_unlock _pthread_self _pthread_setschedparam _pthread_setspecific _putchar _puts _qsort _realloc _sched_yield _sinf _snprintf _strcmp _strdup _strerror _strlen _strncpy _sysconf _sysctlbyname _tanhf _vDSP_maxv _vDSP_sve _vDSP_vadd _vDSP_vdiv _vDSP_vmul _vDSP_vsmul _vDSP_vsub _vfprintf _vm_allocate _vm_deallocate _vsnprintf dyld_stub_binder _ggml_new_tensor_impl _ggml_add_impl _ggml_add1_impl _ggml_acc_impl _ggml_sub_impl _ggml_mul_impl _ggml_div_impl _ggml_scale_impl _ggml_set_impl _ggml_soft_max_impl _ggml_rope_impl _ggml_upscale_impl _ggml_add_rel_pos_impl _ggml_map_binary_impl_f32 _ggml_calloc _ggml_recompute_graph_node _ggml_threadpool_new_impl _ggml_graph_compute_thread _gguf_fread_str _gguf_free_kv _gguf_write_to_buf _ggml_vec_dot_f32 _ggml_vec_dot_f16 _ggml_vec_dot_bf16 _ggml_visit_parents _ggml_add_or_set _ggml_sub_or_set _ggml_graph_compute_secondary_thread _ggml_compute_forward_dup _ggml_compute_forward_diag_mask_f32 _ggml_compute_forward_rope_f16 _ggml_compute_forward_rope_f32 _gguf_bwrite_str _ggml_gallocr_hash_get _alloc_tensor_range _ggml_gallocr_allocate_node __ZN21ggml_backend_registry16register_backendEP16ggml_backend_reg __ZN21ggml_backend_registry15register_deviceEP19ggml_backend_device __ZL37ggml_backend_cpu_buffer_type_get_nameP24ggml_backend_buffer_type __ZL41ggml_backend_cpu_buffer_type_alloc_bufferP24ggml_backend_buffer_typem __ZL42ggml_backend_cpu_buffer_type_get_alignmentP24ggml_backend_buffer_type __ZL36ggml_backend_cpu_buffer_type_is_hostP24ggml_backend_buffer_type __ZL29ggml_backend_cpu_reg_get_nameP16ggml_backend_reg __ZL37ggml_backend_cpu_reg_get_device_countP16ggml_backend_reg __ZL31ggml_backend_cpu_reg_get_deviceP16ggml_backend_regm __ZL34ggml_backend_multi_buffer_get_nameP19ggml_backend_buffer __ZL30ggml_backend_sched_split_graphP18ggml_backend_schedP11ggml_cgraph __ZL21graph_copy_dup_tensor13ggml_hash_setPP11ggml_tensorP12ggml_contextS4_S1_ __ZL22graph_copy_init_tensorP13ggml_hash_setPP11ggml_tensorPbS2_ __ZN21ggml_backend_registryC1Ev __ZN21ggml_backend_registryD1Ev __ZNKSt3__16vectorIP16ggml_backend_regNS_9allocatorIS2_EEE20__throw_length_errorB7v160006Ev __ZNSt3__120__throw_length_errorB7v160006EPKc __ZNSt12length_errorC1B7v160006EPKc __ZSt28__throw_bad_array_new_lengthB7v160006v __ZNKSt3__16vectorIP19ggml_backend_deviceNS_9allocatorIS2_EEE20__throw_length_errorB7v160006Ev __ZL32ggml_backend_cpu_buffer_get_nameP19ggml_backend_buffer __ZL35ggml_backend_cpu_buffer_free_bufferP19ggml_backend_buffer __ZL32ggml_backend_cpu_buffer_get_baseP19ggml_backend_buffer __ZL37ggml_backend_cpu_buffer_memset_tensorP19ggml_backend_bufferP11ggml_tensorhmm __ZL34ggml_backend_cpu_buffer_set_tensorP19ggml_backend_bufferP11ggml_tensorPKvmm __ZL34ggml_backend_cpu_buffer_get_tensorP19ggml_backend_bufferPK11ggml_tensorPvmm __ZL34ggml_backend_cpu_buffer_cpy_tensorP19ggml_backend_bufferPK11ggml_tensorPS1_ __ZL29ggml_backend_cpu_buffer_clearP19ggml_backend_bufferh __ZL25ggml_backend_cpu_get_nameP12ggml_backend __ZL21ggml_backend_cpu_freeP12ggml_backend __ZL40ggml_backend_cpu_get_default_buffer_typeP12ggml_backend __ZL34ggml_backend_cpu_graph_plan_createP12ggml_backendPK11ggml_cgraph __ZL32ggml_backend_cpu_graph_plan_freeP12ggml_backendPv __ZL35ggml_backend_cpu_graph_plan_computeP12ggml_backendPv __ZL30ggml_backend_cpu_graph_computeP12ggml_backendP11ggml_cgraph __ZN31ggml_backend_cpu_device_contextD1Ev __ZL32ggml_backend_cpu_device_get_nameP19ggml_backend_device __ZL39ggml_backend_cpu_device_get_descriptionP19ggml_backend_device __ZL34ggml_backend_cpu_device_get_memoryP19ggml_backend_devicePmS1_ __ZL32ggml_backend_cpu_device_get_typeP19ggml_backend_device __ZL33ggml_backend_cpu_device_get_propsP19ggml_backend_deviceP22ggml_backend_dev_props __ZL28ggml_backend_cpu_device_initP19ggml_backend_devicePKc __ZL39ggml_backend_cpu_device_get_buffer_typeP19ggml_backend_device __ZL39ggml_backend_cpu_device_buffer_from_ptrP19ggml_backend_devicePvmm __ZL35ggml_backend_cpu_device_supports_opP19ggml_backend_devicePK11ggml_tensor __ZL37ggml_backend_cpu_device_supports_buftP19ggml_backend_deviceP24ggml_backend_buffer_type __ZN31ggml_backend_cpu_device_contextC2Ev __ZL37ggml_backend_multi_buffer_free_bufferP19ggml_backend_buffer __ZL31ggml_backend_multi_buffer_clearP19ggml_backend_bufferh __ZL38ggml_backend_sched_backend_id_from_curP18ggml_backend_schedP11ggml_tensor __ZL35ggml_backend_sched_buffer_supportedP18ggml_backend_schedP11ggml_tensori _make_qx_quants _iq2_compare_func _iq3_compare_func _quantize_row_iq3_xxs_impl _quantize_row_iq4_nl_impl _make_qp_quants _iq2_find_best_neighbour _iq1_sort_helper _iq1_find_best_neighbour2 _ggml_backend_metal_buffer_type_get_name _ggml_backend_metal_buffer_type_alloc_buffer _ggml_backend_metal_buffer_type_get_alignment _ggml_backend_metal_buffer_type_get_max_size _ggml_backend_metal_buffer_type_is_host _ggml_metal_init _ggml_backend_metal_buffer_get_name _ggml_backend_metal_buffer_free_buffer _ggml_backend_metal_buffer_get_base _ggml_backend_metal_buffer_set_tensor _ggml_backend_metal_buffer_get_tensor _ggml_backend_metal_buffer_cpy_tensor _ggml_backend_metal_buffer_clear _ggml_backend_metal_name _ggml_backend_metal_free _ggml_backend_metal_get_default_buffer_type _ggml_backend_metal_graph_compute _ggml_backend_metal_supports_op _ggml_backend_metal_supports_buft ___ggml_metal_graph_compute_block_invoke _ggml_metal_encode_node _ggml_metal_supports_op _ggml_metal_get_buffer ___clang_call_terminate __ZL22ggml_backend_blas_nameP12ggml_backend __ZL22ggml_backend_blas_freeP12ggml_backend __ZL41ggml_backend_blas_get_default_buffer_typeP12ggml_backend __ZL31ggml_backend_blas_graph_computeP12ggml_backendP11ggml_cgraph __ZL29ggml_backend_blas_supports_opP12ggml_backendPK11ggml_tensor __ZL31ggml_backend_blas_supports_buftP12ggml_backendP24ggml_backend_buffer_type __ZNKSt3__16vectorINS_6futureIvEENS_9allocatorIS2_EEE20__throw_length_errorB7v160006Ev __ZNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEED1Ev __ZNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEED0Ev __ZNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEE16__on_zero_sharedEv __ZNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEE9__executeEv __ZNSt3__114__thread_proxyB7v160006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMNS_19__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEEFvvEPSF_EEEEEPvSK_ __ZNSt3__110unique_ptrINS_5tupleIJNS0_INS_15__thread_structENS_14default_deleteIS2_EEEEMNS_19__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEEFvvEPSE_EEENS3_ISI_EEED1B7v160006Ev _ggml_backend_register.cold.1 _ggml_backend_device_register.cold.1 _ggml_backend_reg_count.cold.1 _ggml_backend_dev_count.cold.1 __ZN21ggml_backend_registryC1Ev.cold.1 _ggml_hash_size.primes _reltable.gguf_type_name _GGUF_TYPE_SIZE _iq2xxs_grid _ksigns_iq2xs _iq2xs_grid _iq2s_grid _iq3xxs_grid _iq3s_grid _iq1s_grid _kvalues_iq4nl _keven_signs_q2xs _iq2xs_init_impl.kgrid_2bit_256 _iq2xs_init_impl.kgrid_2bit_512 _iq2xs_init_impl.kgrid_1bit_2048 _iq2xs_init_impl.kgrid_2bit_1024 _iq3xs_init_impl.kgrid_256 _iq3xs_init_impl.kgrid_512 _get_scale_shuffle_k4.k_shuffle _get_scale_shuffle.k_shuffle __ZTSNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEE GCC_except_table71 GCC_except_table73 GCC_except_table75 GCC_except_table76 GCC_except_table78 GCC_except_table79 GCC_except_table85 GCC_except_table91 GCC_except_table96 GCC_except_table99 GCC_except_table126 GCC_except_table129 GCC_except_table143 GCC_except_table154 GCC_except_table155 GCC_except_table159 GCC_except_table7 GCC_except_table11 GCC_except_table16 GCC_except_table17 GCC_except_table18 _type_traits _GGML_OP_NAME _GGML_OP_SYMBOL _GGML_UNARY_OP_NAME __ZL18ggml_backend_cpu_i __ZL34ggml_backend_cpu_buffer_from_ptr_i __ZL27ggml_backend_multi_buffer_i __ZL25ggml_backend_cpu_buffer_i __ZL25ggml_backend_cpu_device_i _ggml_backend_metal_i ___block_descriptor_49_e8_v16?0Q8l __ZL14blas_backend_i __ZTVNSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEE __ZTINSt3__119__async_assoc_stateIvNS_12__async_funcIZL25ggml_backend_blas_mul_matP25ggml_backend_blas_contextP11ggml_tensorE3$_0JEEEEE __OBJC_METACLASS_RO_$_GGMLMetalClass __OBJC_CLASS_RO_$_GGMLMetalClass __dyld_private _g_logger_state.0 __ZZ20ggml_backend_cpu_regE20ggml_backend_cpu_reg __ZZL21ggml_backend_cpu_guidvE4guid _ggml_backend_metal_buffer_type.ggml_backend_buffer_type_metal _ggml_backend_metal_buffer_i _ggml_backend_metal_guid.guid __ZZL22ggml_backend_blas_guidvE4guid _g_state _ggml_init.is_first_call _ggml_table_gelu_f16 _ggml_table_gelu_quick_f16 _g_logger_state.1 _g_state_critical __ZZ28ggml_backend_cpu_buffer_typeE28ggml_backend_cpu_buffer_type __ZGVZ28ggml_backend_cpu_buffer_typeE28ggml_backend_cpu_buffer_type __ZZL7get_regvE3reg __ZGVZL7get_regvE3reg __ZZL31ggml_backend_cpu_reg_get_deviceP16ggml_backend_regmE3ctx __ZGVZL31ggml_backend_cpu_reg_get_deviceP16ggml_backend_regmE3ctx __ZZL31ggml_backend_cpu_reg_get_deviceP16ggml_backend_regmE23ggml_backend_cpu_device __ZGVZL31ggml_backend_cpu_reg_get_deviceP16ggml_backend_regmE23ggml_backend_cpu_device __ZZL8fmt_sizemE6buffer _iq2_data _iq3_data _g_backend_device_ref_count _g_backend_device
