• Home
• io.anserini
• anserini
• 1.0.0
• source code
• trec_eval-macosx-aarch64

×

Project download

Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance.
Project price only 1 $

You can buy this project and download/modify it how often you want.

trec_eval.trec_eval-macosx-aarch64 Maven / Gradle / Ivy

????
X? H__PAGEZERO
?
__TEXT
@@__text__TEXT
|f
?__stubs__TEXT?x

?
?x
?__cstring__TEXT$z

??$z
__const__TEXT?>
 ?>__unwind_info__TEXT?>
,
?>?__DATA_CONST@
@@@
__got__DATA_CONST@
(
@!?__DATA?
@?@__data__DATA?
0?__bss__DATA0?
?
H__LINKEDIT?

???

4??83?8??8?	?, VPxxl?%?+F /usr/lib/dyld^HF?
?1??kjk???2 





4*(?8'
/usr/lib/libSystem.B.dylib&??@
)8???????o??{????
?q@???
?
@???
?u@???(
?!???(
??
??#?????????????C????????????(?Ҩ???????????????{@??????(???????(????????^?	?	?}@??|?}@??Y?????
 ?R?Y??X?
?
??
??w???^??^???B`,?#
?c??Y?????@?1aT
o?@??q????HT?@?
?J:?Iy??
	?
??C@???C?^?@?
@?(
?
A???(
???!?,?^Y??RSY???Q?????O?@?

@???????
k?
T
?@??@? 
@?
@???(
???!-?GY? ?R
(
??F???\?)?i?8)q?
T
(
??F???\?	
	?H
?R(
9??\?????
(
?	?F???\?)

??
9?7??F???
?\?
@9?
4
?7@???7?
?\?A
?RI????????7@?	??}	?}@?I??????
????

??Z???(
??F???
?\?
@9?4

?\?
?9	?R??4
?\?	
?9?R))q?? 
T
?\?
?9?H?q????
?@??6
?\????????\?
?9)q?T
?\????????Z??Z?!??Z?A??
??c
????
kAT
?@?
@??Z???Z?
	?	??
ɚ???(
???!?6??H???????Z?a???????Z???Z?
	???
š?7???Z??7@?
}@??c@?q????
?@??6
?@????????@???^?(
?
?@?
?9?>??5
?@????????@?
?9)q?T
??????@?	???
9
?@?	
?9?R))q?? 
T
?@?
?9?>?q????
?@??6
?@????????@??^?(
?
?@?
?9?>??5
?@????????@?
?9)q?T
??????@?	???
9
?@?	
?9?R))q?? 
T
?@?
?9?>?q????
?@??6
?@????????@??@?(
?
?@?
?9?>??5
?@????????@?
?9)q?T
????a?@?	???
9
?@?	
?9?R))q?? 
T
?@?
?9d>?q????
?@??6
?@????????@?
?9)q?T
??????@??@?(
?
?@?
?9J>??5
?@????????@?
?9)q?T
?@?	???
9
?@?	
?9?R))q?? 
T
?@?
?90>?q????
?@??6
?@????????@?
?9)q?T
????
?@?	???
9?@??_?(
????
??_??{E???
??_??
??{????
??F?(
?

?????F??=???_???

??F?(
?

????F??=??_???

??F?(
?

????F??=??@???

??F?(
?

????F??=??@???

??F?(
?

????F??=??@???
 ?R?{C??
??_?????{?????????@?
@??@?

@??=????@??4
?@?????@?@??@?
@??=????@??4
?@?????@?	@??@?
	@??=????
??_??{B?????_????{????
?a7?????
@??#???????????
??A??^?

@??=? 
5
??]?

?!`7??ҟ=????o??^?
@?`??(??=? 
5
?^?
	@?`??(??=?@
4
?#@?
@???!?:?l=?????Y?^?
@??=?#?????
????A??]?

?!`8?K??@??
???
????E
??A??^?

@??]???6=??^?
@??????^?	@?????[?
@????
???G???\?

?!?8?
??,??@???
?

???G???\?

?!?8??? ??@????
??????
?]???\?
	?JT
??[?	@??]?)?|?	ii?
?
G??]?k?{?Ii+???[?)	@??]?)
? 	@?G??]?	?	?
?]???????
?G???\?
}@????cp:?6=???\??_?)%@?
	??T
?_?%@????
??
?]???\?
	??
T
?]??	
?)G??]?)
?(	?
?]???????
?G???\?
}@????c???
k?T
????2???2??????????Q?????
?\??_@?)
@?
	?jT
??[???mG??\?
??)}
?
ii?6??4
??	mG??\?
??k}
?)ik????mG??\?)}
?
	?@?????(QG??(Q??[???mG??\?
??)}
?
	?
@??5??
4
??mG??\?
??)}
?
	?@????Z????


?\???????????UG???QG???! y)?
?@?????????
?@???^?
	??T
?^???@??_??@?J}?)

?)@?yi??" `?
T
?_??@?
??)}
?
	?@??" ` T
?^???@??_??@?J}?)

?)@?yi??" `?T
?_??@?
??)}
?
	?@??" `mT
??
@?
@??_??@?
??)}
?
	?
@???(
???!L??0???????_??@???)}?
	?@??^?@??_??@?J}?)

?)@?y)??@????
?@???^??R)

???*T
?_??@?
??)}
?
	?@??_??@?)}
?
	?
@??0??R?? 5
?_??@?
??)}
?
	?@??_??@?)}
?
	?
@? !????
?@??6
?@???????
?@???^??R)

???jT
?_??@?
??)}
?
	?@??_??@?)}
?
	?
@??0??R
k????
?@??6
?@?@??_??@???J}?)

?)@?yi??_??@?J}?)

?)@?	
	?(?R(
9?@???????
?@????H???@????
k?T
?????^??@????
k?T
????(?R???
??_??{E???
??_??C??{?????????????????]?
@?????^??#??_?@??7???\??7@??????#@??????< 
?=?@?(	???
?@???^?
	?T
?_??@?
??)}
?
	?@??7@? !T
?_??@???)}?
	?@??7???\?	@??#@?
	??#???\?a??????\??_??@?J}?

?@??????#@?????= 
?=?@?(	?
?_??@?
??)}
?
	?@???\?)	@???\??@?K???(y*?
?@?????????]??]????
k?T
????(?R???
??_??{H??C??_?????{?????????_?
@???(
??????/???_?

@???_?	@???*
?(??????/???
?@???_?)
@?
	??
T
??_?@??@?
??)}
?
	???
?@?????????_??????{B?????_?????{
??C???iG?
?
T
?????A??/??@???i????=?(?
?
???G?
?

T
??UG?q/?????
???G?
?

T
??}G?f/?????
??yG?
?

T
??uG?[/???y?
???G?
?

T
??aG?P/?????
???G?
?

T
??eG?E/?????
???G?
?

T
???G?:/?????
???G?
?

T
???G?//?????
???G?
?

T
???G?$/?????
??qG?
?

T
??mG?/???q?
???G?
?

T
???G?/?????
???G?
?

T
???G?/?????
???G?
?

T
???G??.?????
 ?R?{A?????_???
??{??C
????????^?
@?????^?@????_???_???_?!??_?%??_?)??_?-??_?1??_?5??_?9???
?@??_?)@?
	?J#T
?_?
@??@?
??)}
?
	?	@???
?@??_?)
@??@???J}?)

?)	@??_?J
@??@?k}?J
?J@?)
?
	??
T
?@?
@??^?
	?kT

?@?!???????_?
@??@?
??)}
?ii??" `?T
?@??_?)
@??@?
??k}
?)
?)	@?
	?	
??
ɚ?_?(9@?
?(9??_?
@??@?)}
?
	?
@??_?(5@?

?(5??@??_?)
@??@?
??k}
?)
?)	@?
	?	
??
ɚ?_?(1@?
?(1??_?
@??@?)}
?
	?
@??_?(-@?

?(-?
?@????
?@??_?)@?
	??T
?_?
@??@?
??)}
?
	?	@???
?@??_?)
@??@???J}?)

?)	@??_?J
@??@?k}?J
?J@?)
?
	??T
?_?
@??@?
??)}
?
	?	@???
?@??_?)
@??@???J}?)

?)	@??_?J
@??@?k}?J
?J@?)
?
	??T
?@??@?)
@?yi??@?)
@?		?(
@y(
y?@?
@??@?)
@?
	??
T
??
@?
@??@?
@???(
???!?
?.?????e?@?
@??@?)
@?
	??T
?@?
@??^?
	??T
?_?(@??(??@?
@??^?
	??T
?_?(!@??(!??_?()@??()?

?@?
@??^?
	??T
?_?(@??(??@?
@??^?
	??T
?_?(%@??(%??_?()@??()?



?@?!??????
?@?!???k??
?@????W??
?@????????_?
@??_?(@?

?(??_?
!@??_?(%@?

?(%?(?R???
??_??{E???
??_?????{?????????_?
@???_?	@???_?)	@?}	??@?
?R??n-???_?
@?????
?@???_?)	@?
	?"T
?@???_?)@??@?(y*???_?		@??@?
	???
?@????????{B?????_?????{
??C????????G??_?		@?H??}	??_?)	@?}	???!?=?#??L??@??? ?
?????G??_?		@?H??}	???!?=?
??=??@?????
???????(?G??#???
?G??'??_?J	@??+???=????^?
???
?????????
?@??@?
	?T
?@??@?)?|?ii???^?
@??@?
??)}
?i)??@??@?	?@??^?
@??@?)}
?
	?	??^?
@??@?)}
?
	???@????
?@?????????
?@???^?)@?
	?*
T
??
?@??@??R)

????
T
?^?
@??@?
??)}
?ii??@?
	?????
?@??6

?@????????@??@?
	??
T
??^?@??@?yi??^?)
@??@???J}?)

?(??@??^?)
@??@???J}?(i*??@??a^?^?
@??@?)}?
	?	???^?@??@?yi??^?)
@??@?J}?)

?(??@????

?@????????^?
@??@?
}@????c  ????^?	???
?@??@?
	?JT
?^?
@??@?
??)}
?
	?
@??^?(	@?

?(	?
?@????????@??^?(?(?R???
??_??{E???
??_??????????
?@??@?)@?
	?JT
?@??@?)
@??@???J}?)ij?
	?a
T
?@?
@??@?
??)}
?
	?	@???	
?@????????/??
?@?????_??C??????@?	@??@?
	@?8ax?C??_?????{
?????????????????_???^??^?????????
k?T
????I???/????
?@??@?
	??T
?7@??@?yi??_?)!@?
	?KT
?@?????@?
??@?
	?
b?!?
@`n
a?@?(a???_?	@?
?
T
?@??@??@???

?	?	?`?0/?$?


?@????????@??
?
?@??+@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{J?????_????o??{?????????????????^???]??]???4????
k?T
????e
??\????#?f
??
k?T
????Z
?/?_?????????K@???[??[@?H
??
?G@??[@?
??)}
?
	?	@??C??/?C?
?C@?????W?n???S?
?S@???X??R)

????T
??V? `?ן??
?@??6
??[??S@?yi??#??
??????W??W?(a???W@???W?
?[@?	?R?{??
??
?W@??G@??[@???J}?)

?)@?
	??ן?{?
?{@?6
(???W??[@???[??[@?H
??
?G@??[@?
??)}
?
	?	@??;??/?;?
?;@?????????V? `?T
??V??V?(a??
??V??U? a?T
??V???U?(a????U???U?(a???
??W? ` T
??W??7???U?`(a?W?8aI?
@`?7@?
a?_@?(a?_?
?^?	@?
?
T
?S@??[@???W??W???V??V???U??_@????3?

?	?	?
???
??`??
?@?

?S@???S?p??
?S@???X?
	??T
??[??S@?yi??#?w
??????W??W?(a???U???U?(a?????W? ` T
??W??/???U?`(a?W?8a?
@`?/@?
a?_@?(a?_?
?^?	@?
?T
?S@??[@???W??W??V??_@???

?	?	?
??/?
??`?p???
?S@???S????
??V? `M
T
?W@???W?
?[@?	?R?W??
??
?W@??G@??[@???J}?)

?)@?
	??ן?W?
?W@?6
(???W??[@???[??[@?H
??
?G@??[@?
??)}
?
	?	@??'??/?'?
?'@?????????V? `?T
??V??V?(a??
?^?	@?
?mT
?S@??[@??W???V??V???

?	??/	?
?
??`????
?S@???S?????V? `?
T
?_@??V?a?\?	@???\?)%@?	??
?G@?k?(?R???
??^??{W??oV????_???
??{??C
?????????????_?!@?h
?
?_?!@?	@????_?!@?@???
??^?@??@?
	?	??}	?}@?[??^?
???
?????????
?@??@?
	?T
?@??@?)?|?ii???^?
@??@?
??)}
?i)??@??@?	?@??^?
@??@?)}
?
	?	??^?
@??@?)}
?
	???@????
?@?????????
?@???^?)@?
	?*
T
??
?@??@??R)

????
T
?^?
@??@?
??)}
?ii??@?
	?????
?@??6

?@????????@??@?
	??
T
??^?@??@?yi??^?)
@??@???J}?)

?(??@??^?)
@??@???J}?(i*??@??a^?^?
@??@?)}?
	?	???^?@??@?yi??^?)
@??@?J}?)

?(??@????

?@????????^?
@??@?
}@????cP????^?	???
?@??@?
	?JT
?^?
@??@?
??)}
?
	?
@??^?(	@?

?(	?
?@????????@??^?(?(?R???
??_??{E???
??_??????????
?@??@?)@?
	?JT
?@??@?)
@??@???J}?)ij?
	?a
T
?@?
@??@?
??)}
?
	?	@???	
?@????????/??
?@?????_??C??????@?	@??@?
	@?8ax?C??_????{??????????????????]?!@?	@?????????_???^??^????????
k?T
????w?'@???
???r???
??[??@?
	??T
??[???\??\?)yj?
	??T
?@??a^????[??'@?
	?T
?'@?????[???
?@??@?

b?a?]?	@???]?)%@??\?)

?	???\??????]?)!@?)@?
	?aT

?3@???[?yi??_?)!@?
	??T
?@????

??[????????
?\???]?)!@?)@?
	?T
??\??\?yi?????@??a^????[??'@?
	?T
?'@?????[???
?@??@?

b?a?]?	@???]?)%@??\?)

?	???\???????(?R???
??_??{K????_?????{	??C???????????????]?!@?	@?????????_???^??^??#?????
k?T
????T?'?
?'@??@?
	?JT
?'@???\??\?)yj?
	?AT
?@?
?/n
?a?]?	@???]?)%@??\?)

?	???\??????]?)!@?)@?
	?aT

?#@??'@?yi??_?)!@?
	??T
?@????

?'@???'????
?\???]?)!@?)@?
	?jT
?@?
?/n
?a?]?	@???]?)%@??\?)

?	???\???????(?R???
??_??{I?????_?????{
?????????????????/???_???^??^?????????
k?T
????{????????
?@??@?
	??T
?7@??@?yi??aT
S?7@??@?yi?	??T
?@????I?7@??@?yi??
??
?7@??@?yi??_?)!@?
	??T
?@????7?@?????@???
?@?
n(a??+?@??a^???@?n(c``?@??@?B(c!b?@???@?
	??@?
	?
b??@?Bc!b?@??
b???YG?B(c?@???@?
	?
b???eG?c(dBc!b
(a?@?(a??


?@????????+@?
?
?+@?
?a^?@?a??
?@??]?	@???]?)%@?	??(?R???
??_??{J?????_?????{
??????????????????]?!@?	@?????????/???_???^??^??c?????
k?T
????`?@???
???[???
??[??@?
	?JT
??[???\??\?)yj?
	?!T
?@??@?!?a^a?]?	@???]?)%@??\?)

?	???\??????]?)!@?)@?
	?aT

?+@???[?yi??_?)!@?
	??
T
?@?????@??a^??[??

b?
a?@?(a??

??[????????
?\???]?)!@?)@?
	?JT
?@??@?!?a^a?]?	@???]?)%@??\?)

?	???\???????(?R???
??_??{J?????_?????{
??C????????????????/???_???^??^??
??????
k?T
??????/?
?/@??_?)!@?
	?
T
?K@??/@?	yi??@?
	???
?/@???/?????+??'??#??/?
?/@??7@?
	?*T
?O@??/@?yi??aT
M?O@??/@?yi?	??T
?#@???#?C?O@??/@?yi??
??
?O@??/@?yi??_?)!@?
	??T
?+@???+?1?'@???'??+@?
?T
?+@??C@?
	?T
?C@????+@???
?@?
b????@??C@?
	?T
?C@????@???
?@??@?

b?
an
8a?@?(a???@?
n(a??


?/@???/?????C@?
?
?C@?
?a^?@?a??
?@???
YG? a?T
?@?????YG???
?@????]?	@???]?)%@?	???_?	@??
T
?@??]?	@???]?)%@?	?@???

??`?H'???
(?R???
??_??{M?????_???????????????@?@???(
? ?R????_??
??{?????????????_?@??
?@?

@???
@???K?k??+
?*?(	?`??(???
 ?R?{C??
??_????{??????????????????]?!@?	@?????_???^??^?????????
k?T
????h??\?
@?x??#@?
	?T
?#@?????\?
@?x???
?@????@?
?T
??
????@??@?#?????!@?????@????
????A??
?@??@?
	?jT
?;@??@?yi????@?%?T
??R?9?@????
?@????9?@???T
??R?9?@?	??T
??R?9??R?9



?@9??)@??@?)

?(
9
?@?????????@??@?
	?
9(?R???
??_??{K????_??
??{????????????@?	@??_?)%@?)?|?ii??@?

@???@???+
?*?(	?`?\1??
? ?R?{C??
??_?????{?????????????	@?
?

T
??	???@??
?
?@?!@?
@?H?
?@?	@??@?)%@?)?|?ii??
??@?!@?
@??
??@?!@?	@??
?
 ?R?{B?????_????{???
??????????????_???^??^????????
k?T
?????@??
?
?@??a^?@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{G????_????{???
??????????????_???^??^????????
k?T
?????@??
?
?@??a^?@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{G????_??C??{????????????????_???^??^??C??????
k?T
????#?@???
?@?h?
?@??a^???@??@?
	?T
?@????@???
?@??@?

b?a?]?	@???]?)%@?	??
(?R???
??_??{H??C??_????{???
??????????????_???^??^???X????
k?T
?????@???
?@???
?@??a^?@?!?a^a?@?!?a^?@?B?a^!ba?]?	@???]?)%@?	??
(?R???
??_??{G????_?????{	??C???????????????]?!@?	@?????_???^??^??c?!????
k?T
????*?@???
?@??a^?@?!?a^a???@??a^?@?!?a^a????\?
@????@?
n(a?@?a?@?a?@??@?!b?@?!(ba?]?	@???]?)%@?	??
(?R???
??_??{I?????_????{???
??????????????_???^??^????????
k?T
?????@??a^?@?!?a^8a?@?!?a^8a?@?!?a^8a?]?	@???]?)%@?	??(?R???
??_??{G????_??C??{????????????????_???^??^??c??????
k?T
????2????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	????@??@?)}
?
	?	%@??@?
	????@??@?)}
?
	?	)@??@?
	???
?@????????@??a^?]?	@???]?)%@?	??(?R???
??_??{H??C??_??C??{????????????????_???^??^??c?W????
k?T
????*????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	????@??@?)}
?
	?	!@??@?
	???
?@????????@??a^?]?	@???]?)%@?	??(?R???
??_??{H??C??_??C??{????????????????_???^??^??c?????
k?T
????"????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	???
?@????????@??a^?]?	@???]?)%@?	??(?R???
??_??{H??C??_??C??{????????????????_???^??^??c??????
k?T
????J??????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	????@??@?)}
?
	?	!@??@?
	????@??@?)}
?
	?	@??@?
	????@??@?)}
?
	?	%@??@?
	????@??@?)}
?
	?	)@??@?
	???
?@????????@??
?
?@??a^?@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{H??C??_?????{	??C??????????????/?????_???^??^???y????
k?T
???????
?@??#@?
	??T
?@????
?@??#@?
	?
T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@?????+@??@?yi??@?	?
@y
c?+@??@?yi??@?yix?+@??@?)yj??@?)yjx
	

b
a?@?(a??

?@???????
?@?????????
?@??#@?
	?T
?#@???
?@??@?
	?
T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@?????+@??@?yi??@?	?
@y
c?+@??@?yi??@?yix?+@??@?)yj??@?)yjx
	

b
a?@?(a??

?@???????
?@????????#@???
?@??@?
	??T
?@????
?@??@?
	?*T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@????

?@???????
?@????????@??
?
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_?????{	??C??????????????_???^??^????????
k?T
????R?/????
?@??@?
	?
T
?@??@?
??)}
?
	?@????@??@?)}
?
	?	!@??@?
	????@??@?)}
?
	?@????@??@?)}
?
	?	%@??@?
	????@??@?)}
?
	?	)@??@?
	????@?h
?
?@??a^?@?!?a^
a?@?(a??

?@????????@? `?
T
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_?????{
?????????????????_???^??^????5????
k?T
??????/????
?@??@?
	?*T
?#@??@?
??)}
?
	?5@????#@??@?)}
?
	?-@????@??@?
	?	T
?#@??@?
??)}
?
	?@????#@??@?)}
?
	?	!@??@?
	????#@??@?)}
?
	?@????#@??@?)}
?
	?	%@??@?
	????#@??@?)}
?
	?	)@??@?
	????#@??@?)}
?
	?9@??@??@?)
?	}	??@?
	????#@??@?)}
?
	?5@??@??@?)

?	}	??@?
	????@??a^?@?!?a^
a?@?(a???#@??@?
??)}
?
	??+@??C??#?%??@??a^?@?!?a^
a?@?(a??

?@????????@? `?
T
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{J?????_?????O??K??G??C?????;??O@?@?
?-T
?/?;??O@?5@??#??O@?
@??O@?)@?)?
??)}
?
	?	@??'??7?
?7@??O@?)@?
	?
T
?7@???3?
?3@??O@?)@?)?
	?T
?O@?
@??7@?
??)}
?
	?	@??/?
?/@??O@?)
@??7@???J}?)

?)	@??O@?J
@??7@?k}?J
?J@?)
?
	?T
?O@?
@??3@?
??)}
?
	?	@??+?
?+@??O@?)
@??3@???J}?)

?)	@??O@?J
@??3@?k}?J
?J@?)
?
	?T
?/@?
@??+@?)
@?
	?
T
?/@?
@??K@?
	??T
??@??????;@???;?

?+@?!??+????
?/@?!??/????
?3@???3????
?7@???7?????7?
?7@??O@?)@?
	?j	T
?O@?
@??7@?
??)}
?
	?	@??/?
?/@??O@?)
@??7@???J}?)

?)	@??O@?J
@??7@?k}?J
?J@?)
?
	?bT
?'@??+?
?+@??'@??#@?)
?
	?T
?/@?
@??+@?)
@?
	?
T
?/@?
@??K@?
	??T
??@??????;@???;?

?+@?!??+????
?/@?!??/????
?7@???7??????@??;@?
	??;?&
?O@?
@??????O@?@?????
?@??@?
	?jT
?O@?@??@?yi??" `?
T
?@?????@??O@?)5@?)?
	?aT


?@????????@?????
?@??K@?
	?JT
?@??@?
	?k
T
?O@?@??@?yi??" `aT
w??
?@??@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
?;@???;?

?@????????@????
?@??K@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
??@?????

?@????????K@???
?@??@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
??@?????

?@???????
?@????t???K@???
?@??@?
	?JT
?@??@?
	?k
T
?O@?@??@?yi??" `aT
O??
?@??K@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
?;@???;?

?@????????K@???
?@??@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
?;@???;?

?@???????
?@?????????@??;@?
	??;?
??@??G@?(
??;@??C@?(
?????_??C??{????????????????_???^??^??c??????
k?T
????2??????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	????@??@?)}
?
	?	@??@?
	???
?@????????@??
?
?@??a^?@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{H??C??_?????{	??C??????????????/?????_???^??^???7????
k?T
????U??
?@??#@?
	??T
?@????
?@??#@?
	?
T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@?????+@??@?yi??@?	?
@y
c?+@??@?yi??@?yix?+@??@?)yj??@?)yjx
	

b
a?@?(a??

?@???????
?@????????@??
?
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_?????{	??C??????????????_???^??^????????
k?T
????:?/????
?@??@?
	?
T
?@??@?
??)}
?
	?@????@??@?)}
?
	?@????@?h
?
?@??a^?@?!?a^
a?@?(a??

?@????????@? `?
T
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_?????{
?????????????????_???^??^????z????
k?T
????s?/????
?@??@?
	?*T
?#@??@?
??)}
?
	?9@????#@??@?)}
?
	?1@????@??@?
	?T
?#@??@?
??)}
?
	?@????#@??@?)}
?
	?@????#@??@?)}
?
	?9@??@??@?)
?	}	??@?
	????#@??@?)}
?
	?5@??@??@?)

?	}	??@?
	????@??a^?@?!?a^
a?@?(a???#@??@?
??)}
?
	??+@??C??#?%??@??a^?@?!?a^
a?@?(a??

?@????????@? `?
T
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{J?????_?????O??K??G??C?????;??O@?@?
?T
?/?;??O@?9@??#??O@?
@??O@?)@?)?
??)}
?
	?	@??'??7?
?7@??O@?)@?
	??T
?7@???3?
?3@??O@?)@?)?
	??
T
?O@?
@??7@?
??)}
?
	?	@??/?
?/@??O@?
@??7@???J}?

?	@??O@?J
@??7@?k}?J
?J@?

??R)

???"
T
?/@?
@??K@?
	?觟??
?@?H6
?O@?
@??3@?
??)}
?
	?	@??+?
?+@??O@?
@??3@???J}?

?	@??O@?J
@??3@?k}?J
?J@?

??R)

???"
T
?+@?
@??K@?
	?觟??
?@??6
?/@?
@??+@?)
@?
	??T
??@??????;@???;?

?+@?!??+????
?/@?!??/????
?3@???3????
?7@???7????7?
?7@??O@?)@?
	?T
?O@?
@??7@?
??)}
?
	?	@??/?
?/@??O@?
@??7@???J}?

?	@??O@?J
@??7@?k}?J
?J@?

??R)

???"
T
?/@?
@??K@?
	?觟??
?@?(6
?'@??+?
?+@??'@??#@?

??R)

???"
T
?+@?
@??K@?
	?觟??
?@??6
?/@?
@??+@?)
@?
	??T
??@??????;@???;?

?+@?!??+????
?/@?!??/????
?7@???7??????@??;@?
	??;???O@?
@???????
?@??K@?
	?jT
?O@?@??@?yi??" `?
T
?@?????@??O@?)5@?)?
	?aT


?@????????@?????
?@??K@?
	?jT
?@??@?
	?k
T
?O@?@??@?yi??" `aT
P??
?@??@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
?;@???;?

?@????????@????
?@??K@?
	?*T
?@??@?
	?k
T
?O@?@??@?yi??" `aT
?@??@?yi??@?
	?
@9?4
??@?????

?@???????
?@?????????@??;@?
	??;?
??@??G@?(
??;@??C@?(
?????_??C??{????????????????_???^??^??c?N????
k?T
????B??????
?@??@?
	?
T
?@??@?
??)}
?
	?	@??@?
	????@??@?)}
?
	?	!@??@?
	????@??@?)}
?
	?	@??@?
	????@??@?)}
?
	?	%@??@?
	???
?@????????@??
?
?@??a^?@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{H??C??_?????{	??C??????????????/?????_???^??^????????
k?T
???????
?@??#@?
	??T
?@????
?@??#@?
	?
T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@?????+@??@?yi??@?	?
@y
c?+@??@?yi??@?yix?+@??@?)yj??@?)yjx
	

b
a?@?(a??

?@???????
?@?????????
?@??#@?
	?T
?#@???
?@??@?
	?
T
?+@??@?yi??@?yix(
5
?+@??@?yi??@?yix?4
?@?????+@??@?yi??@?	?
@y
c?+@??@?yi??@?yix?+@??@?)yj??@?)yjx
	

b
a?@?(a??

?@???????
?@????????@??
?
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_?????{	??C??????????????_???^??^???D????
k?T
????J?/????
?@??@?
	?
T
?@??@?
??)}
?
	?@????@??@?)}
?
	?	!@??@?
	????@??@?)}
?
	?@????@??@?)}
?
	?	%@??@?
	????@?h
?
?@??a^?@?!?a^
a?@?(a??

?@????????@? `?
T
?@??@?!?a^a?]?	@???]?)%@?	??
(?R???
??_??{I?????_????{???
??????????????_???^??^???!????
k?T
????\??
?@??@?
	?T
???/????
?@??@??@?)
?)@?
	??T
?@??@?	?@??@?yi??_?)!@?
	??
T
?@?????@??a^?@??

b?
a?@?(a??

?@????????@?H?
?@??@??@?	?
@?!?a^
a?]?	@???]?)%@?	?@?(a?

?@????????@??
T
?@?
?a^?]?	@???]?)%@?	?@?a?
(?R???
??_??{G????_?????{
??????????????????]?!@?	@?????_???^??^??
??????
k?T
???????]?!@?@??}?}@?'?????
???????
?@??'@?
	?
T
??
?@???]?)!@?)@?
	??T
??\??@?yi??+@??@?	?
@?!?a^ah?
aG?(ax??\??@?(y*?
?@????????+@??@?)?z?ii??a^?+@??@?	?
@?!?a^a???@?????]?!@?@???
??[?	?R???
??
?\???[?yi??+@??@?)
?)@?
	??ן??
?@?H6
?+@??@?)?z?ii??a^?\???[?
yi?!?a^
a?]?	@???]?)%@???[?)

?	?@?(a???[???????+@??@?)?z?ii????+@??@?	?@???
?@??R)
????T
?@?
??ן??
?@??6
?@??a^?@?!?a^a???@??@? a?T
?@???

??[?	?R??H
??
?@??\???[?)yj?
	?????
?@?(6
?@??]?	@???]?)%@???[?)

?	?@?(a???[???????+@??@?	?@??@?)?yi??_?)!@?
	??T
?@????

?@???????
?@????>???'@??T
???
??[???]?)!@?)@?
	?jT
?'@?
?a^?]?	@???]?)%@???[?)

?	?@?a?
??[????????
?\?&
?(?R???
??_??{J?????_????{???
???????????????]?!@?	@????_???^??^??#??????
k?T
???????
?@??@?
	?T
??????
?@??@??@?)
?)@?
	??T
?@??@??@?)yj?
	?T
?@??a^?@?!?a^
a?]?	@???]?)%@??@?)

?	?@?(a??@??????]?)!@?)@?
	?aT

?@??@?	?@??@?yi??_?)!@?
	??T
?@????

?@???????
?@???]?)!@?)@?
	??T
?@??a^?@??@?
yi?!?a^
a?]?	@???]?)%@??@?)

?	?@?(a??@???????
?@????????@??T
??
?@???]?)!@?)@?
	?jT
?@?
?a^?]?	@???]?)%@??@?)

?	?@?a?
?@???????
(?R???
??_??{G????_?????{	??C??????????????_???^??^????????
k?T
????8???/????
?@??@?
	?T
?/@??@?yi??_?)!@?
	??
T
?@?????@??a^?@??

b?
a?@?(a??

?@????????@?
n (`?#@?B?a^!(b?@?!8baA??]?	@???]?)%@?	??(?R???
??_??{I?????_?p?@??p?@??p?
@??p?@??p?@??p?@??p?"@??p?&@??p?*@??p?.@??p?2@??p?6@??p?:@??p?>@??p?B@??p?F@??p?J@??p?N@??p?R@??p?V@??p?Z@??p?^@??p?b@??p?f@??p?j@??p?n@??p?z@??p?~@??p??@??p??@??p??@??p??@??p??@??qrelstrec_resultshelpversionquery_eval_wantedmeasurecomplete_rel_info_wantedlevel_for_relnosummaryDebug_levelJudged_docs_onlyNumber_docs_in_collMax_retrieved_per_topicRel_info_formatResults_formatOutput_old_results_formatZscorehvqm:cl:nD:JN:M:R:T:oZ:trec_eval version %s
trec_eval: illegal measure '%s'
trec_eval: Quit in file '%s'
trec_eval: Illegal rel_format '%s'
trec_eval: Illegal retrieval results format '%s'
officialtrec_eval: illegal measure 'official'
alltrec_eval: Cannot initialize measure '%s'
trec_eval: Can't calculate measure '%s'
trec_eval: Can't accumulate measure '%s'
trec_eval: Can't print query measure '%s'
trec_eval: No queries with both results and relevance info
trec_eval: Can't print measure '%s'
trec_eval: cleanup failed
9.0.4Usage: trec_eval [-h] [-q] {-m measure}* trec_rel_file trec_top_file
   -h: Give full help information, including other options
   -q: In addition to summary evaluation, give evaluation for each query
   -m: calculate and print measures indicated by 'measure'
       ('-m all_qrels' prints all qrels measures, '-m official' is default)
trec_eval: improper measure in parameter '%s'
%s
-----------------------
Individual measure documentation for requested measures
%s
%s-- No measures indicated.
   Request measure documentation using <-m measure> on command line
trec_eval [-h] [-q] [-m measure[.params] [-c] [-n] [-l ]
   [-D debug_level] [-N ] [-M ] [-R rel_format] [-T results_format]
   rel_info_file  results_file 
 
Calculate and print various evaluation measures, evaluating the results  
in results_file against the relevance info in rel_info_file. 
 
There are a fair number of options, of which only the lower case options are 
normally ever used.   
 --help:
 -h: Print full help message and exit. Full help message will include
     descriptions for any measures designated by a '-m' parameter, and
     input file format descriptions for any rel_info_format given by '-R'
     and any top results_format given by '-T.'
     Thus to see all info about preference measures use
          trec_eval -h -m all_prefs -R prefs -T trec_results 
 --version:
 -v: Print version of trec_eval and exit.
 --query_eval_wanted:
 -q: In addition to summary evaluation, give evaluation for each query/topic
 --measure measure_name[.measure_params]:
 -m measure: Add 'measure' to the lists of measures to calculate and print.
    If 'measure' contains a '.', then the name of the measure is everything
    preceeding the period, and everything to the right of the period is
    assumed to be a list of parameters for the measure, separated by ','. 
    There can be multiple occurrences of the -m flag.
    'measure' can also be a nickname for a set of measures. Current 
    nicknames include 
       'official': the main measures often used by TREC
       'all_trec': all measures calculated with the standard TREC
                   results and rel_info format files.
       'set': subset of all_trec that calculates unranked values.
       'prefs': Measures not in all_trec that calculate preference measures.
 --complete_rel_info_wanted:
 -c: Average over the complete set of queries in the relevance judgements  
     instead of the queries in the intersection of relevance judgements 
     and results.  Missing queries will contribute a value of 0 to all 
     evaluation measures (which may or may not be reasonable for a  
     particular evaluation measure, but is reasonable for standard TREC 
     measures.) Default is off.
 --level_for_rel num:
 -l: Num indicates the minimum relevance judgement value needed for 
      a document to be called relevant. Used if rel_info_file contains 
      relevance judged on a multi-relevance scale.  Default is 1. 
 --nosummary:
 -n: No summary evaluation will be printed
 --Debug_level num:
 -D : Debug level.  1 and 2 used for measures, 3 and 4 for merging
     rel_info and results, 5 and 6 for input.  Currently, num can be of the
     form . and only qid will be evaluated with debug info printed.
     Default is 0.
 --Number_docs_in_coll num:
 -N : Number of docs in collection Default is MAX_LONG 
 -Max_retrieved_per_topic num:
 -M : Max number of docs per topic to use in evaluation (discard rest). 
      Default is MAX_LONG.
 --Judged_docs_only:
 -J: Calculate all values only over the judged (either relevant or  
     nonrelevant) documents.  All unjudged documents are removed from the 
     retrieved set before any calculations (possibly leaving an empty set). 
     DO NOT USE, unless you really know what you're doing - very easy to get 
     reasonable looking numbers in a file that you will later forget were 
     calculated  with the -J flag.  
 --Rel_info_format format:
 -R format: The rel_info file is assumed to be in format 'format'.  Current
    values for 'format' include 'qrels', 'prefs', 'qrels_prefs'.  Note not
    all measures can be calculated with all formats.
 --Results_format format:
 -T format: the top results_file is assumed to be in format 'format'. Current
    values for 'format' include 'trec_results'. Note not all measures can be
    calculated with all formats.
 --Zscore Zmean_file:
 -Z Zmean_file: Instead of printing the raw score for each measure, print
    a Z score instead. The score printed will be the deviation from the mean
    of the raw score, expressed in standard deviations, where the mean and
    standard deviation for each measure and query are found in Zmean_file.
    If mean is not in Zmeanfile for a measure and query, -1000000 is printed.
    Zmean_file format is ascii lines of form 
       qid  measure_name  mean  std_dev
 
 
Standard evaluation procedure:
For each of the standard TREC measures requested, a ranked list of
of relevance judgements is created corresponding to each ranked retrieved doc,
A rel judgement is set to -1 if the document was not in the pool (not in 
rel_info_file) or -2 if the document was in the pool but unjudged (some 
measures (infAP) allow the pool to be sampled instead of judged fully).  
Otherwise it is set to the value in rel_info_file. 
Most measures, but not all, will treat -1 or -2 the same as 0, 
namely nonrelevant.  Note that relevance_level is used to 
determine if the document is relevant during score calculations. 
Queries for which there is no relevance information are ignored. 
Warning: queries for which there are relevant docs but no retrieved docs 
are also ignored by default.  This allows systems to evaluate over subsets  
of the relevant docs, but means if a system improperly retrieves no docs,  
it will not be detected.  Use the -c flag to avoid this behavior. 
Rel_info_file format: Standard 'qrels'
Relevance for each docno to qid is determined from rel_info_file, which 
consists of text tuples of the form 
   qid  iter  docno  rel 
giving TREC document numbers (docno, a string) and their relevance (rel,  
a non-negative integer less than 128, or -1 (unjudged)) 
to query qid (a string).  iter string field is ignored.   
Fields are separated by whitespace, string fields can contain no whitespace. 
File may contain no NULL characters. 
qrels_jgRel_info_file format: Standard 'qrels'
Relevance for each docno to qid is determined from rel_info_file, which 
consists of text tuples of the form 
   qid  ujg  docno  rel 
giving TREC document numbers (docno, a string) and their relevance (rel,  
a non-negative integer less than 128, or -1 (unjudged)) 
to query qid (a string) for a particular user judgment group. 
This allows averaging (or other operations) of appropriate evaluation measures
across multiple users, whoc may differ in their judgments. 
Fields are separated by whitespace, string fields can contain no whitespace. 
File may contain no NULL characters. 
prefsRel_info_file format: Non-standard 'prefs'
Preferences of user(s) for docs for a given qid is determined from
text_prefs_file, which consists of text tuples of the form
   qid  ujg  ujsubg  docno  rel_level
giving TREC document numbers (docno, a string) and their relevance
level (rel_level,a non-negative float) to query qid (a string) for a 
user judgment sub-group (ujsubg, a string) within a user judgment
group (ujg, a string).
Fields are separated by whitespace, string fields can contain no whitespace.
File may contain no NULL characters.

Preferences are indicated indirectly by comparing rel_level of
different docnos within the same user judgment sub group(JSG).  A
judgment sub group establishes preferences between all docnos with
non-tied rel_levels within the group. Except possibly for 0.0, the
actual values of rel_level are ignored by default; they only serve to
establish a ranking within the JSG.

If a user only expresses a preference between two docs, then that user JSG
will have 2 lines in text_prefs_file:
      qid1  ujg1  sub1 docno1  3.0
      qid1  ujg1  sub1 docno2  2.0

If a user completely ranks some small number N (5-10) of docs, then N lines 
are used.
For example:
      qid1  ujg1  sub1  docno1  3.0
      qid1  ujg1  sub1  docno2  2.0
      qid1  ujg1  sub1  docno3  0.0
      qid1  ujg1  sub1  docno4  6.0
      qid1  ujg1  sub1  docno5  0.0
      qid1  ujg1  sub1  docno6  2.0
establishes a total of 13 preferences (5 with docno4 preferred, 4 with docno1 
preferred, 2 each with docno2 and docno6 preferred).

If a given user has multiple preferences that aren't complete, the preferences
are expressed in multiple JSGs within a single JG.
For example:
      qid1  ujg1  sub1  docno1  3.0
      qid1  ujg1  sub1  docno2  2.0
      qid1  ujg1  sub1  docno3  1.0
      qid1  ujg1  sub2  docno1  2.0
      qid1  ujg1  sub2  docno2  1.0
      qid1  ujg1  sub2  docno4  3.0
expressses 5 preferences (1>2, 1>3, 2 > 3, 4>1, 4>2).  Note the duplicate
1 > 2 is not counted as a separate preference

Multiple users are indicated by different JGs.
For example:
      qid1  ujg1  sub1  docno1  3.0
      qid1  ujg1  sub1  docno2  2.0
      qid1  ujg2  sub1  docno1  0.0
      qid1  ujg2  sub1  docno3  6.0
      qid1  ujg2  sub1  docno4  2.0
      qid1  ujg2  sub2  docno1  0.0
      qid1  ujg2  sub2  docno2  8.0
expressses 5 preferences (1>2, 3>1, 4>1, 3>4, 2>1).

A Judgment Group (JG) conceptually represents preferences for a single
information need of a user at a single time.  Within a single JG, it
is an error if there are inconsistencies (doc A > doc B in one JSG,
but B > A or B == A in another).  The different JSGs within a JG are
just a mechanism tha allows expressing partial ordering within a JG.
Within a single JG, preferences are transistive:
      qid1  ujg1  sub1  docno1  3.0
      qid1  ujg1  sub1  docno2  2.0
      qid1  ujg1  sub1  docno3  1.0
      qid1  ujg1  sub2  docno2  5.0
      qid1  ujg1  sub2  docno4  4.0
expresses 5 preferences (1>2, 1>3, 2>3, 2>4, 1>4).  There is no
preference expressed between 3 and 4.

Different JGs may contain contradictory preferences, as in an earlier
example.  These disagreements are realistic and desirable: users (or
even the same user at different times) often do not agree with each
other's preferences.  Individual preference evaluation measures will
handle these contradictions (or confirmations) in different ways.

A rel_level of 0.0 by convention means that doc is non-relevant to the
topic (in that user's opinion).  it is an inconsistency (and an error)
if a doc is assigned a rel_level of 0.0 in one JSG, but a different
rel_level value in another JSG of the same JG.  Some preference
evaluation measures may handle 0.0 differently.  Thus when converting
a preference file in some other format into text_prefs format, do not
assign a rel_level of 0.0 to a docno unless it is known that docno was
considered nonrelevant.

Handling of rel_level 0.0 separately addresses the general problem
that the number of nonrelevant docs judged for a topic can be critical
to fair evaluation - adding a couple of hundred preferences involving
nonrelevant docs (out of the possibly millions or billions in a
collection) can both change the importance of the topic when averaging
and even change whether system A scores better than system B on a
topic (even given identical retrieval on the added nonrel docs).  How
to handle this correctly for preference evaluation will be an
important future research problem.
qrels_prefsRel_info_file format: Non-standard 'qrels_prefs'
The file format is exactly the same as rel_info_file format 'qrels',
however it is interpreted as a restricted 'prefs' rel_info_file.
It cannot represent some user preferences (in particular, if a single user
prefers Doc A to Doc B, and A to C, but does not express a preference
between A and C) , but it allows the standard TREC qrels file to serve as 
input for preference evaluation measures.

Read all relevance preference information from text_qrels_prefs_file.
Preferences of user(s) for docs for a given qid is determined from
text_prefs_file, which consists of text tuples of the form
   qid  ujg   docno  rel_level
giving TREC document numbers (docno, a string) and their relevance
level (rel_level,a non-negative float) to query qid (a string) for a 
 user judgment group (ujg, a string).
Fields are separated by whitespace, string fields can contain no whitespace.
File may contain no NULL characters.

Preferences are indicated indirectly by comparing rel_level of
different docnos within the same user judgment group(JG).  A
judgment group establishes preferences between all docnos with
non-tied rel_levels within the group. Except possibly for 0.0, the
actual values of rel_level are ignored by default; they only serve to
establish a ranking within the JSG.

If a user only expresses a preference between two docs, then that user JSG
will have 2 lines in text_prefs_file:
      qid1  ujg1   docno1  3.0
      qid1  ujg1   docno2  2.0

If a user completely ranks some small number N (5-10) of docs, then N lines 
are used.
For example:
      qid1  ujg1    docno1  3.0
      qid1  ujg1    docno2  2.0
      qid1  ujg1    docno3  0.0
      qid1  ujg1    docno4  6.0
      qid1  ujg1    docno5  0.0
      qid1  ujg1    docno6  2.0
establishes a total of 13 preferences (5 with docno4 preferred, 4 with docno1 
preferred, 2 each with docno2 and docno6 preferred).

A Judgment Group (JG) conceptually represents preferences for a single
information need of a user at a single time.  Within a single JG, it
is an error if there are inconsistencies (doc A > doc B in one JSG,
but B > A or B == A in another).

Different JGs may contain contradictory preferences, These
disagreements are realistic and desirable: users (or even the same
user at different times) often do not agree with each other's
preferences.  Individual preference evaluation measures will handle
these contradictions (or confirmations) in different ways.

A rel_level of 0.0 by convention means that doc is non-relevant to the
topic (in that user's opinion).  Some preference evaluation measures
may handle 0.0 differently.  Thus when converting a preference file in
some other format into text_prefs format, do not assign a rel_level of
0.0 to a docno unless it is known that docno was considered
nonrelevant.

Handling of rel_level 0.0 separately addresses the general problem
that the number of nonrelevant docs judged for a topic can be critical
to fair evaluation - adding a couple of hundred preferences involving
nonrelevant docs (out of the possibly millions or billions in a
collection) can both change the importance of the topic when averaging
and even change whether system A scores better than system B on a
topic (even given identical retrieval on the added nonrel docs).  How
to handle this correctly for preference evaluation will be an
important future research problem.
Results_file format: Standard 'trec_results'
Lines of results_file are of the form 
     030  Q0  ZF08-175-870  0   4238   prise1 
     qid iter   docno      rank  sim   run_id 
giving TREC document numbers (a string) retrieved by query qid  
(a string) with similarity sim (a float).  The other fields are ignored, 
with the exception that the run_id field of the last line is kept and 
output.  In particular, note that the rank field is ignored here; 
internally ranks are assigned by sorting by the sim field with ties  
broken deterministicly (using docno). 
Sim is assumed to be higher for the docs to be retrieved first. 
File may contain no NULL characters. 
Lines may contain fields after the run_id; they are ignored. 
Process for evaluating qrels and trec_resultsProcess for evaluating qrels_jg and trec_resultsProcess for evaluating prefs and trec_results   Copyright (c) 2008 - Chris Buckley. 

   Permission is granted for use and modification of this file for
   research, non-commercial purposes. 

   Process for evaluating qrels_prefs and trec_resultstrec_eval: Negative cutoff detected
trec_eval: duplicate cutoffs detected
trec_eval: malformed pair parameters in '%s'
%s_%ld%s_%3.2f%s_%sZ%-22s	%s	%6.4f
%-22s	%s	%6.4f
%-22s	%s	%ld
%s%-22s	%s	%6.4f
Ztrec_eval.get_qrels: Cannot read qrels file '%s'
trec_eval.get_qrels: Malformed line %ld
trec_eval.get_results: Cannot read results file '%s'
trec_eval.get_results: Cannot copy results file '%s'
trec_eval.get_results: Cannot close results file '%s'
trec_eval.get_results: Malformed line %ld
trec_eval.get_prefs: Cannot read prefs file '%s'
trec_eval.get_prefs: Malformed line %ld
trec_eval.get_qrels_prefs: Malformed line %ld
0trec_eval.get_qrels_jg: Malformed line %ld
trec_eval.get_qrels: duplicate docs %s
trec_eval.form_res_qrels: rel_info format not qrels or results format not trec_results
trec_eval.form_res_qrels: duplicate docs %strec_eval.form_res_rels: duplicate docs %s
no_queryno querytrec_eval: rel_info format not qrels_jg or results format not trec_results
Debug: Form_prefs starting query '%s'
Returned Cached Form_prefs
trec_eval.form_prefs_info: prefs_info format not (prefs or qrels_prefs) or results format not trec_results
trec_eval.form_prefs_counts: Internal docid %ld occurs with different rel_level in same jsg
trec_eval.form_prefs_counts: doc '%s' has both 0 and non-0 rel_level assigned
trec_eval.form_prefs_counts: Pref inconsistency found
      internal rank %ld and internal rank %ld are conflicted
After input, before rankstrec_eval.form_prefs_counts: duplicate docs %sAfter -M, ranksInput, before ranksAfter marking not judgedAfter assigning docid_ranksForm_prefs: num_judged %ld, num_judged_ret %ld
Final prefsDocno_results Dump.  num_results %ld, %s
  %s	%4.2f	%3ld
Prefs_and_ranks Dump.  num_pref_lines %ld,  %s
  %s	%s	%4.2f	%s	%3ld
Results_prefs Dump.  %ld Judgment Groups
  num_judged_ret %ld,  num_judged %ld
  JG Dump.  Type %s
ECPrefs_array    num_prefs_fulfilled_ret %ld
    num_prefs_possible_ret %ld
    num_prefs_fulfilled_imp %ld
    num_prefs_possible_imp %ld
    num_prefs_possible_notoccur %ld
    num_nonrel %ld
    num_nonrel_ret %ld
    num_rel %ld
    num_rel_ret %ld
    Rel_array Dump. %ld values
      %4.2f     JG is not initialized (0 ECs and no rel_array
    Dump of %ld ECs within JG
    Prefs_Array Dump. Num_judged %ld
      Row %3ld
          (%ld)
     %2hhd    EC Dump. Rel_level %4.2f. Num_docid_ranks %ld%3ld   Counts_Array Dump. Num_judged %ld
    Row %3ld
        (%ld)
   %2hd trec_eval.get_zscores: Cannot read zscores file '%s'
trec_eval.get_zscores: Malformed line %ld
setall_trecall_prefsrunidnum_qnum_retnum_relnum_rel_retmapgm_mapRprecbprefrecip_rankiprec_at_recallPutilityset_Pset_recallset_relative_Pset_mapset_FrelstringrecallinfAPgm_bpref11pt_avgndcgrelative_PRprec_multsuccessmap_cutndcg_cutndcg_relRndcgbinGGnum_nonrel_judged_retprefs_num_prefs_possprefs_num_prefs_fulprefs_num_prefs_ful_retprefs_simpprefs_pairprefs_avgjgprefs_avgjg_Rnonrelprefs_simp_retprefs_pair_retprefs_avgjg_retprefs_avgjg_Rnonrel_retprefs_simp_impprefs_pair_impprefs_avgjg_impmap_avgjgP_avgjgRprec_mult_avgjg    Mean Average Precision
    Precision measured after each relevant doc is retrieved, then averaged
    for the topic, and then averaged over topics (if more than one).
    This is the main single-valued number used to compare the entire rankings
    of two or more retrieval methods.  It has proven in practice to be useful
    and robust.
    The name of the measure is unfortunately inaccurate since it is 
    calculated for a single topic (and thus don't want both 'mean' and
    'average') but was dictated by common usage and the need to distiguish
    map from Precision averaged over topics (I had to give up my attempts to
    call it something else!)
    History: Developed by Chris Buckley after TREC 1.
    Cite: 'Retrieval System Evaluation', Chris Buckley and Ellen Voorhees.
    Chapter 3 in TREC: Experiment and Evaluation in Information Retrieval
    edited by Ellen Voorhees and Donna Harman.  MIT Press 2005
    Precision at cutoffs
    Precision measured at various doc level cutoffs in the ranking.
    If the cutoff is larger than the number of docs retrieved, then
    it is assumed nonrelevant docs fill in the rest.  Eg, if a method
    retrieves 15 docs of which 4 are relevant, then P20 is 0.2 (4/20).
    Precision is a very nice user oriented measure, and a good comparison
    number for a single topic, but it does not average well. For example,
    P20 has very different expected characteristics if there 300
    total relevant docs for a topic as opposed to 10.
    Note:   trec_eval -m P.50 ...
    is different from 
            trec_eval -M 50 -m set_P ...
    in that the latter will not fill in with nonrel docs if less than 50
    docs retrieved
    Cutoffs must be positive without duplicates
    Default param: -m P.5,10,15,20,30,100,200,500,1000
    Number of topics results averaged over.  May be different from
    number of topics in the results file if -c was used on the command line 
    in which case number of topics in the rel_info file is used.
    Number of documents retrieved for topic. 
    May be affected by Judged_docs_only and Max_retrieved_per_topic command
    line parameters (as are most measures).
    Summary figure is sum of individual topics, not average.
    Number of relevant documents for topic. 
    May be affected by Judged_docs_only and Max_retrieved_per_topic command
    line parameters (as are most measures).
    Summary figure is sum of individual topics, not average.
trec_eval: m_num_rel: rel_info format not qrels or qrels_jg
    Number of relevant documents retrieved for topic. 
    May be affected by Judged_docs_only and Max_retrieved_per_topic command
    line parameters (as are most measures).
    Summary figure is sum of individual topics, not average.
    Geometric Mean Average Precision
    This is the same measure as 'map' (see description of 'map') on an
    individual topic, but the geometric mean is calculated when averaging
    over topics.  This rewards methods that are more consistent over topics
    as opposed to methods which do very well for some topics but very poorly
    for others.
    gm_ap is reported only in the summary over all topics, not for individual
    topics.
    Precision after R documents have been retrieved.
    R is the total number of relevant docs for the topic.  
    This is a good single point measure for an entire retrieval
    ranking that averages well since each topic is being averaged
    at an equivalent point in its result ranking.
    Note that this is the point that Precision = Recall.
    History: Originally developed for IR rankings by Chris Buckley
    after TREC 1, but analogs were used in other disciplines previously.
    (the point where P = R is an important one!)
    Cite: 'Retrieval System Evaluation', Chris Buckley and Ellen Voorhees.
    Chapter 3 in TREC: Experiment and Evaluation in Information Retrieval
    edited by Ellen Voorhees and Donna Harman.  MIT Press 2005
    Reciprocal Rank of the first relevant retrieved doc.
    Measure is most useful for tasks in which there is only one relevant
    doc, or the user only wants one relevant doc.
    Main binary preference measure.
    Fraction of the top R nonrelevant docs that are retrieved after each
    relevant doc. Put another way: when looking at the R relevant docs, and
    the top R nonrelevant docs, if all relevant docs are to be preferred to
    nonrelevant docs, bpref is the fraction of the preferences that the
    ranking preserves.
    Cite: 'Retrieval Evaluation with Incomplete Information', Chris Buckley
    and Ellen Voorhees. In Proceedings of 27th SIGIR, 2004.
    Interpolated Precision at recall cutoffs.
    This is the data shown in the standard Recall-Precision graph.
    The standard cutoffs and interpolation are needed to average data over
    multiple topics; otherwise, how is a topic with 5 relevant docs averaged
    with a topic with 3 relevant docs for graphing purposes?  The Precision 
    interpolation used here is
      Int_Prec (rankX) == MAX (Prec (rankY)) for all Y >= X.
    Default usage: -m iprec_at_recall.0,.1,.2,.3,.4,.5,.6,.7,.8,.9,1 ...
    Recall at cutoffs
    Recall (relevant retrieved / relevant) measured at various doc level
    cutoffs in the ranking. If the cutoff is larger than the number of docs
    retrieved, then it is assumed nonrelevant docs fill in the rest.
    REcall is a fine single topic measure, but does not average well.
    Cutoffs must be positive without duplicates
    Default param: -m recall.5,10,15,20,30,100,200,500,1000
    Precision measured at multiples of R (num_rel).
    This is an attempt to measure topics at the same multiple milestones
    in a retrieval (see explanation of R-prec), in order to determine
    whether methods are precision oriented or recall oriented.  If method A
    dominates method B at the low multiples but performs less well at the
    high multiples then it is precision oriented (compared to B).
    Default param: -m Rprec_mult.0.2,0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.0 ...
    Set utility measure
    Set evaluation based on contingency table:
                        relevant  nonrelevant
       retrieved            a          b
       nonretrieved         c          d
    where  utility = p1 * a + p2 * b + p3 * c + p4 * d
    and p1-4 are parameters (given on command line in that order).
    Conceptually, each retrieved relevant doc is worth something positive to
    a user, each retrieved nonrelevant doc has a negative worth, each 
    relevant doc not retrieved may have a negative worth, and each
    nonrelevant doc not retrieved may have a (small) positive worth.
    The overall measure is simply a weighted sum of these values.
    If p4 is non-zero, then '-N num_docs_in_coll' may also be needed - the
    standard results and rel_info files do not contain that information.
    Default usage: -m utility.1.0,-1.0,0.0,0.0 ...
    Warning: Current version summary evaluation averages over all topics;
    it could be argued that simply summing is more useful (but not backward
    compatible)
trec_eval.calc_utility: improper number of coefficients
    Interpolated Precision averaged over 11 recall points
    Obsolete, only use for comparisons of old runs; should use map instead.
    Average interpolated at the given recall points - default is the
    11 points being reported for ircl_prn.
    Both map and 11-pt_avg (and even R-prec) can be regarded as estimates of
    the area under the standard ircl_prn curve.
    Warning: name assumes user does not change default parameter values:
    measure name is independent of parameter values and number of parameters.
    Will actually average over all parameter values given.
    To get 3-pt_avg as in trec_eval version 8 and earlier, use
      trec_eval -m 11-pt_avg.0.2,0.5,0.8 ...
    Default usage: -m 11-pt_avg.0.0,.1,.2,.3,.4,.5,.6,.7,.8..9,1.0
trec_eval.calc_m_11ptavg: No cutoff values
    Normalized Discounted Cumulative Gain
    Compute a traditional nDCG measure according to Jarvelin and
    Kekalainen (ACM ToIS v. 20, pp. 422-446, 2002)
    Gain values are set to the appropriate relevance level by default.  
    The default gain can be overridden on the command line by having 
    comma separated parameters 'rel_level=gain'.
    Eg, 'trec_eval -m ndcg.1=3.5,2=9.0,4=7.0 ...'
    will give gains 3.5, 9.0, 3.0, 7.0 for relevance levels 1,2,3,4
    respectively (level 3 remains at the default).
    Gains are allowed to be 0 or negative, and relevance level 0
    can be given a gain.
    Based on an implementation by Ian Soboroff
ndcg: %ld %ld %3.1f %6.4f %3.1f %6.4f
    Normalized Discounted Cumulative Gain at cutoffs.
    Compute a traditional nDCG measure according to Jarvelin and
    Kekalainen (ACM ToIS v. 20, pp. 422-446, 2002) at cutoffs.
    See comments for ndcg.
    Gain values are the relevance values in the qrels file.  For now, if you
    want different gains, change the qrels file appropriately.
    Cutoffs must be positive without duplicates
    Default params: -m ndcg_cut.5,10,15,20,30,100,200,500,1000
    Based on an implementation by Ian Soboroff
ndcg_cut: cutoff %ld dcg %6.4f
ndcg_cut:%ld %3.1f %6.4f
ndcg_cut: cutoff %ld idcg %6.4f
ndcg_cut:%ld %ld %3.1f %6.4f
    Normalized Discounted Cumulative Gain at R levels
    Experimental measure
    Compute a traditional nDCG measure according to Jarvelin and
    Kekalainen (ACM ToIS v. 20, pp. 422-446, 2002), averaged at the various
    R level points. The R levels are the number of docs at each non-negative
    gain level in the judgments, with the gain levels sorted in decreasing
    order. Thus if there are 5 docs with gain_level 3, 3 with gain 2, 10
    with gain 1, and 50 with gain 0, then 
    Rndcg = 1/4 (ndcg_at_5 + ndcg_at_8 + ndcg_at_18 + ndcg_at_68).
    In this formulation, all unjudged docs have gain 0.0, and thus there is
    a final implied R-level change at num_retrieved.
    Idea behind Rndcg, is that the expected value of ndcg is a smoothly
    decreasing function, with discontinuities upward at each transistion
    between positive gain levels in the ideal ndcg.  Once the gain level 
    becomes 0, the expected value of ndcg then increases until all docs are
    retrieved. Thus averaging ndcg is problematic, because these transistions
    occur at different points for each topic.  Since it is not unusual for
    ndcg to start off near 1.0, decrease to 0.25, and then increase to 0.75
    at various cutoffs, the points at which ndcg is measured are important.
    
    Gain values are set to the appropriate relevance level by default.  
    The default gain can be overridden on the command line by having 
    comma separated parameters 'rel_level=gain'.
    Eg, 'trec_eval -m Rndcg.1=3.5,2=9.0,4=7.0 ...'
    will give gains 3.5, 9.0, 3.0, 7.0 for relevance levels 1,2,3,4
    respectively (level 3 remains at the default).
    Gains are allowed to be 0 or negative, and relevance level 0
    can be given a gain.
Rndcg: %ld %ld %3.1f %6.4f %3.1f %6.4f %6.4f
Rndcg: %ld %ld %3.1f %6.4f %3.1f %6.4f
    Normalized Discounted Cumulative Gain averaged over rel docs
    Experimental measure
    Compute a traditional nDCG measure according to Jarvelin and
    Kekalainen (ACM ToIS v. 20, pp. 422-446, 2002), averaged at rel docs.
    Idea behind ndcg_rel, is that the expected value of ndcg is a smoothly
    decreasing function, with discontinuities upward at each transistion
    between positive gain levels in the ideal ndcg.  Once the gain level 
    becomes 0, the expected value of ndcg then increases until all rel docs are
    retrieved. Thus averaging ndcg is problematic, because these transistions
    occur at different points for each topic.  Since it is not unusual for
    ndcg to start off near 1.0, decrease to 0.25, and then increase to 0.75
    at various cutoffs, the points at which ndcg is measured are important.
    This version averages ndcg over each relevant doc, where relevant is
    defined as expected gain > 0.  If a rel doc is not retrieved, then
    ndcg for the doc is the dcg at the end of the retrieval / ideal dcg. 
    
    Gain values are set to the appropriate relevance level by default.  
    The default gain can be overridden on the command line by having 
    comma separated parameters 'rel_level=gain'.
    Eg, 'trec_eval -m ndcg_rel.1=3.5,2=9.0,4=7.0 ...'
    will give gains 3.5, 9.0, 3.0, 7.0 for relevance levels 1,2,3,4
    respectively (level 3 remains at the default).
    Gains are allowed to be 0 or negative, and relevance level 0
    can be given a gain.
ndcg_rel: %ld %ld %3.1f %6.4f %3.1f %6.4f %6.4f
ndcg_rel: %ld %ld %3.1f %6.4f %3.1f %6.4f
ndcg_rel: %ld %ld %6.4f %6.4f %6.4f
    Binary G
    Experimental measure. (4/10/2008)
    G is a gain related measure that combines qualities of MAP and NDCG.
    G(doc) == rel_level_gain (doc) / log2 (2+num_nonrel retrieved before doc)
    G is the average of G(doc) over all docs, normalized by
    sum (rel_level_gain).
    BinG restricts the gain to either 0 or 1 (nonrel or rel), and thus is the
    average over all rel docs of (1 / log2 (2+num_nonrel before doc))
binG: %ld %ld %6.4f
    Normalized Gain
    Experimental measure 4/10/2008
    G is a gain related measure that combines qualities of MAP and NDCG.
    Contribution of doc doc retrieved at rank i is 
    G(doc) == gain (doc) / log2 (2+ideal_gain(i)-results_gain(i))
    where results_gain(i) is sum gain(doc) for all docs before i
    and ideal_gain is the maximum possible results_gain(i)
    G is the sum of G(doc) over all docs, normalized by max ideal_gain.
    Gain values are set to the appropriate relevance level by default.  
    The default gain can be overridden on the command line by having 
    comma separated parameters 'rel_level=gain'.
    Eg, 'trec_eval -m G.1=3.5,2=9.0,4=7.0 ...'
    will give gains 3.5, 9.0, 3.0, 7.0 for relevance levels 1,2,3,4
    respectively (level 3 remains at the default).
    Gains are allowed to be 0 or negative, and relevance level 0
    can be given a gain.
    The idea behind G is that the contribution of a doc retrieved at i
    should not be independent of the docs before. If most docs before have
    higher gain, then the retrieval of this doc at i is nearly as good as 
    possible, and should be rewarded appropriately
G: %ld %ld %3.1f %6.4f %3.1f %6.4f %6.4f %6.4f
G: %ld %ld %3.1f %6.4f %3.1f %6.4f %6.4f
G: %ld %ld %3.1f %6.4f %3.1f %6.4f
    Relative Precision at cutoffs
    Precision at cutoff relative to the maximum possible precision at that
    cutoff.  Equivalent to Precision up until R, and then recall after R
    Cutoffs must be positive without duplicates
    Default params: -m relative_P.5,10,15,20,30,100,200,500,1000
    Success at cutoffs
    Success (a relevant doc has been retrieved) measured at various doc level
    cutoffs in the ranking.
    If the cutoff is larger than the number of docs retrieved, then
    it is assumed nonrelevant docs fill in the rest.
    Cutoffs must be positive without duplicates
    Default param: trec_eval -m success.1,5,10
    History: Developed by Stephen Tomlinson.
    Inferred AP
    A measure that allows sampling of judgement pool: Qrels/results divided
    into unpooled, pooled_but_unjudged, pooled_judged_rel,pooled_judged_nonrel.
    My intuition of infAP:
    Assume a judgment pool with a random subset that has been judged.
    Calculate P at rel doc using only the judged higher retrieved docs,
    then average in 0's from higher docs that were not in the judgment pool.
    (Those in the pool but not judged are ignored, since they are assumed
    to be relevant in the same proportion as those judged.)
    Cite:    'Estimating Average Precision with Incomplete and Imperfect
    Judgments', Emine Yilmaz and Javed A. Aslam. CIKM 
    Mean Average Precision at cutoffs
    Map measured at various doc level cutoffs in the ranking.
    If the cutoff is larger than the number of docs retrieved, then
    it is assumed nonrelevant docs fill in the rest.
    Map itself is precision measured after each relevant doc is retrieved,
    averaged over all relevant docs for the topic.
    Cutoffs must be positive without duplicates
    Default param: -m map_cut.5,10,15,20,30,100,200,500,1000
   Binary preference (bpref), but using goemetric mean over topics
    See the explanation for 'bpref' for the base measure for a single topic.
    Gm_bpref uses the geometric mean to combine the single topic scores.
    This rewards methods that are more consistent across topics as opposed to
    high scores for some topics and low scores for others.
    Gm_bpref is printed only as a summary measure across topics, not for the
    individual topics.
gm_bpref: bpref %6.4f, gm_bpref %6.4f    Runid given by results input file.
%-22s	%s	%s
    The relevance values for the first N (default 10) retrieved docs
    are printed as a string, one character per relevance value for a doc.
    If the relevance value is between 0 and 9, it is printed.
    If the value is > 9,  '>' is printed.
    If the document was not in the pool to be judged, '-' is printed.
    if the document was in the pool, but unjudged (eg, infAP),  '.' is printed
    if the document has some other relevance value, '<' is printed.
    Measure is only printed for individual queries.
    Default usage:  -m relstring.10 
%-22s	%s	'%s'
    Set Precision: num_relevant_retrieved / num_retrieved 
    Precision over all docs retrieved for a topic.
    Was known as exact_prec in earlier versions of trec_eval
    Note:   trec_eval -m P.50 ...
    is different from 
            trec_eval -M 50 -m set_P ...
    in that the latter will not fill in with nonrel docs if less than 
    50 docs retrieved
    Set Recall: num_relevant_retrieved / num_relevant 
    Recall over all docs retrieved for a topic.
    Was known as exact_recall in earlier versions of trec_eval    Relative Set Precision:  P / (Max possible P for this size set) 
    Relative precision over all docs retrieved for a topic.
    Was known as exact_relative_prec in earlier versions of trec_eval
    Note:   trec_eval -m relative_P.50 ...
    is different from 
            trec_eval -M 50 -m set_relative_P ...
    Set map: num_relevant_retrieved**2 / (num_retrieved*num_rel)
    Unranked set map, where the precision due to all relevant retrieved docs
    is the set precision, and the precision due to all relevant not-retrieved
    docs is set to 0.
    Was known as exact_unranked_avg_prec in earlier versions of trec_eval.
    Another way of loooking at this is  Recall * Precision on the set of
    docs retrieved for a topic.
      Set F measure: weighted harmonic mean of recall and precision
    set_Fx = (x+1) * P * R / (R + x*P)
    where x is the relative importance of R to P (default 1.0).
    Default usage: trec_eval -m set_F.1.0 ...
    Cite: Variant of van Rijsbergen's E measure ('Information Retrieval',
    Butterworths, 1979).
    Number of non-relevant judged documents retrieved for topic. 
    Not an evaluation number per se, but gives details of retrieval results.
    Summary figure is sum of individual topics, not average.
    Number of possible prefs independent of whether documents retrieved
    Summary figure is sum of individual topics, not average.
    Number of prefs fulfilled
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), and counts as failure if neither A nor B retrieved.
    Summary figure is sum of individual topics, not average.
    Number of prefs fulfilled among retrieved docs
    For doc pref A>B, both A nd B must be retrieved to be counted.
    Summary figure is sum of individual topics, not average.
    Simple ratio of preferences fulfilled to preferences possible.
    If a doc pair satisfies two preferences, both are counted.
    If preferences are conflicted for a doc pair, all are counted
    (and thus max possible score may be less than 1.0 for topic).
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), and counts as failure if neither A nor B retrieved.
    Assumes '-R prefs' or '-R qrels_prefs'
   Average over doc pairs of preference ratio for that pair.
    If a doc pair satisfies 3 preferences but fails 2 preferences (preferences
    from 5 different users),  then the score for doc pair is 3/5.
    Same as prefs_simp if there are no doc_pairs in multiple judgment groups.
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), and counts as failure if neither A nor B retrieved.
    Assumes '-R prefs' or '-R qrels_prefs'
    Simple ratio of preferences fulfilled to preferences possible
    within a judgment group, averaged over jgs.  I.e., rather than considering
    all preferences equal (prefs_simp), consider all judgment groups equal.
    prefs_avgjg = AVERAGE_OVER_JG (fulfilled_jg / possible_jg);
    May be useful in applications where user satisfaction is represented
    by a jg per user, and it is not desirable for many preferences expressed
    by user1 to swamp a few preferences by user2.
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), and counts as failure if neither A nor B retrieved.
    Assumes '-R prefs' or '-R qrels_prefs'
    Ratio of preferences fulfilled to preferences possible within a
    judgment group, averaged over jgs, except that the number of
    nonrelevant retrieved docs (rel_level == 0.0) in each jg is set to
    R, the number of relevant retrieved docs (rel_level > 0.0) in that jg.
    
    This addresses the general problem that the number of
    nonrelevant docs judged for a topic can be critical to fair
    evaluation - adding a couple of hundred preferences involving
    nonrelevant docs (out of the possibly millions in a collection) can
    both change the importance of the topic when averaging and even
    change whether system A scores better than system B (even given
    identical retrieval on the added nonrel docs).
    
    This measure conceptually sets the number of nonrelevant retrieved
    docs of a jg to R. If the actual number, N, is less than R, then R
    * (R-N) fulfilled preferences are added.  If N is greater than R,
    then only the first R (rank order) docs in the single ec with
    rel_level = 0.0 are used and the number of preferences are
    recalculated.  
    If there is a single jg with two equivalence classes (one of them 0.0), 
    then prefs_avgjg_Rnonrel is akin to the ranked measure bpref.
    Assumes '-R prefs' or '-R qrels_prefs'
    Simple ratio of preferences fulfilled to preferences possible among.
    the retrieved docs. 
    If a doc pair satisfies two preferences, both are counted.
    If preferences are conflicted for a doc pair, all are counted
    (and thus max possible score may be less than 1.0 for topic).
    For doc pref A>B, A and B must both be retrieved to be counted as either
    fulfilled or possible.
    pref_*_ret measures should be used for dynamic collections but are
    inferior in most other applications.
    Assumes '-R prefs' or '-R qrels_prefs'
   Average over doc pairs of preference ratio for that pair.
    If a doc pair satisfies 3 preferences but fails 2 preferences (preferences
    from 5 different users),  then the score for doc pair is 3/5.
    Same as prefs_simp if there are no doc_pairs in multiple judgment groups.
    For doc pref A>B, A and B must both be retrieved to be counted as either
    fulfilled or possible.
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), and counts as failure if neither A nor B retrieved.
    pref_*_ret measures should be used for dynamic collections but are
    inferior in most other applications.
    Assumes '-R prefs' or '-R qrels_prefs'
    Simple ratio of preferences fulfilled to preferences possible
    within a judgment group, averaged over jgs.  I.e., rather than considering
    all preferences equal (prefs_simp), consider all judgment groups equal.
    prefs_avgjg = AVERAGE_OVER_JG (fulfilled_jg / possible_jg);
    May be useful in applications where user satisfaction is represented
    by a jg per user, and it is not desirable for many preferences expressed
    by user1 to swamp a few preferences by user2.
    For doc pref A>B, A and B must both be retrieved to be counted as either
    fulfilled or possible.
    pref_*_ret measures should be used for dynamic collections but are
    inferior in most other applications.
    Assumes '-R prefs' or '-R qrels_prefs'
    Ratio of preferences fulfilled to preferences possible within a
    judgment group, averaged over jgs, except that the number of
    nonrelevant retrieved docs (rel_level == 0.0) in each jg is set to
    R, the number of relevant retrieved docs (rel_level > 0.0) in that jg.
    
    This addresses the general problem that the number of
    nonrelevant docs judged for a topic can be critical to fair
    evaluation - adding a couple of hundred preferences involving
    nonrelevant docs (out of the possibly millions in a collection) can
    both change the importance of the topic when averaging and even
    change whether system A scores better than system B (even given
    identical retrieval on the added nonrel docs).
    
    This measure conceptually sets the number of nonrelevant retrieved
    docs of a jg to R. If the actual number, N, is less than R, then R
    * (R-N) fulfilled preferences are added.  If N is greater than R,
    then only the first R (rank order) docs in the single ec with
    rel_level = 0.0 are used and the number of preferences are
    recalculated.  
    If there is a single jg with two equivalence classes (one of them 0.0), 
    then prefs_avgjg_Rnonrel is akin to the ranked measure bpref.
    For doc pref A>B, A and B must both be retrieved to be counted as either
    fulfilled or possible.
    pref_*_ret measures should be used for dynamic collections but are
    inferior in most other applications.
    Assumes '-R prefs' or '-R qrels_prefs'
    Simple ratio of preferences fulfilled to preferences possible.
    If a doc pair satisfies two preferences, both are counted.
    If preferences are conflicted for a doc pair, all are counted
    (and thus max possible score may be less than 1.0 for topic).
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), but ignores pair if neither A nor B retrieved.
    pref_*_imp measures don't have any preferred applications that I know of,
    but some people like them.
    Assumes '-R prefs' or '-R qrels_prefs'
   Average over doc pairs of preference ratio for that pair.
    If a doc pair satisfies 3 preferences but fails 2 preferences (preferences
    from 5 different users),  then the score for doc pair is 3/5.
    Same as prefs_simp if there are no doc_pairs in multiple judgment groups.
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), but ignores pair if neither A nor B retrieved.
    pref_*_imp measures don't have any preferred applications that I know of,
    but some people like them.
    Assumes '-R prefs' or '-R qrels_prefs'
    Simple ratio of preferences fulfilled to preferences possible
    within a judgment group, averaged over jgs.  I.e., rather than considering
    all preferences equal (prefs_simp), consider all judgment groups equal.
    prefs_avgjg = AVERAGE_OVER_JG (fulfilled_jg / possible_jg);
    May be useful in applications where user satisfaction is represented
    by a jg per user, and it is not desirable for many preferences expressed
    by user1 to swamp a few preferences by user2.
    For doc pref A>B, this includes implied preferences (only one of A or B
    retrieved), but ignores pair if neither A nor B retrieved.
    pref_*_imp measures don't have any preferred applications that I know of,
    but some people like them.
    Assumes '-R prefs' or '-R qrels_prefs'
    Mean Average Precision over judgment groups 
    Precision measured after each relevant doc is retrieved, then averaged
    for the topic, and then averaged over judgement group (user) and then 
    averaged over topics (if more than one).
    Same as the workhorse measure 'map' except if there is more than one
    set of relevance judgments for this query (each set indicated by a
    different judgment group), the score will be averaged over the judgment
    groups.
    Precision measured at multiples of R(num_rel) averged over users.
    This is an attempt to measure topics at the same multiple milestones
    in a retrieval (see explanation of R-prec), in order to determine
    whether methods are precision oriented or recall oriented.  If method A
    dominates method B at the low multiples but performs less well at the
    high multiples then it is precision oriented (compared to B).
    If there is more than one judgment group (set of evalutation judgments
    of a user), then the measure is averaged over those jgs.
    Default param: 
    trec_eval -m Rprec_mult_avgjg.0.2,0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.0  ...
    Precision at cutoffs, averaged over judgment groups (users)
    Precision measured at various doc level cutoffs in the ranking.
    If the cutoff is larger than the number of docs retrieved, then
    it is assumed nonrelevant docs fill in the rest.  Eg, if a method
    retrieves 15 docs of which 4 are relevant, then P20 is 0.2 (4/20).
    If there are multiple relevance judgment sets for this query, Precision
    is averaged over the judgment groups.
    Cutoffs must be positive without duplicates
    Default param: trec_eval -m P.5,10,15,20,30,100,200,500,1000
yaap    Yet Another Average Precision
    Adaptation of MAP proposed by Stephen Robertson to get a value
    that is more globally averagable than MAP.  Should be monotonic with
    MAP on a single topic, but handles extreme values better.
    log ((1 + sum_probrel)  /  (1 + num_rel - sum_probrel))
    where sum_probrel = sum over all rels of (numrel_before_it / current rank)
    Cite: 'On Smoothing Average Precision', Stephen Robertson.
    ECIR 2012, LNCS 7224, pp.158-169.  2012.
    Edited by R.Baeza-Yates et al. Springer-Verlag Berlin
?h㈵??>??.??????????h㈵??>
44 0?LL?x
L3?hH
?
+,+
,?,
?,. /
XA?DP????????	???
??
?)??0>t?,C@B@BXA?D0????????p?
??
 (??=?>XA?D0????????j?
~?
 (??=?>XA?DP????????@?{?
k?
?.??0>t?,Ct?,Ct?,CXA?G?????????????????
9?
?3,
?=?>XA?G???????????????????????????333333????????????333333??ffffff??????????????????????
Z?
l8?
?=?>XA?G??@????????	??
?
?)H

0>t?,CXA?G?? ???????? ?
??
l8P
?=?>XA?GП ????????/?
?
 (4(
?=?>XA?D0????????4?
??
l8?)
?=?>XA?G?@????????	???
??)?3
0>t?,Ct?,CXA?DP????????	p??
B?)$:
0>t?,C?A
HB
X?0????????$@??
f (?B
?=?>XA?D0??????????
?
 (?C
?=?>XA?D0??????????
w (@D
?=?>XA?D0??????????
? ((E
?=?>XA?DP????????
p???
Z?3?E
?=?>XA?G?0??????????6?
??(G
?=?>@B@B@B@BXA?D0??????????
L (HL
?=?>XA?D0??????????
$ (?O
?=?>XA?D0??????????
? (dQ
?=?>XA?D0??????????
?  (,\
?=?>XA?D0??????????
?" (P]
?=?>XA?D0??????????
?% (_
?=?>XA?D0??????????
?( (P`
?=?>XA?D0????????	?
g. (i
?=?>XA?D0?????????
?0 (dj
?=?>XA?D0????????'?
?2 ((m
?=?>XA?D0????????7?
?5 (?n
?=?>XA?DP????????
??I?
?7?.xp
0>t?,Ct?,CXA?D???????? h?%
0@@
@?



:
Z
p
?
?
?
?
?
?
?
?
?




*

D

R

d

n

z

?

?

?

?

?

?

?

?



"
0
<
N
^___memcpy_chk___memset_chk___snprintf_chk___stderrp___stdoutp___strncpy_chk_abort_atof_atol_close_exit_exp_fflush_fprintf_fputs_free_getopt_long_index_isspace_log_log2_lseek_malloc_memcpy_memset_mmap_munmap_open_optarg_optind_printf_putchar_qsort_read_realloc_strcmp_strlen
_	_mh_execute_headerform_prefs_counts=mainCte_j??$?|a_cutHempty`se?{?{
acc_meas_Mc?
form_?get_?init_meas_?meas_?num_?print_?re?trec_measure??~a_cut?
empty?
s?
?}?}
_gm?
??
alc_avg_meas_?
hk_and_?onvert_to_zscore?رmalloc?realloc?????ȅ
inter_procs?pref_counts_cleanup?res_rels?????
_??cleanup?jg???
_cleanup?????

_cleanup???
prefs?qrels?trec_results?zscores???

_??
cleanup?jg?prefs???

_cleanup?????

_cleanup???
??

_cleanup?ļ
??
_cleanup????]float?long??Sa_float_cut_?empty?s_??O?P
_p_??gfloat?pair??pfloat?long??Q??
11pt_avg?G?P?R?b?gm_?i?map?n?prefs_?	r?s?utility?yaap???
??

_avgjg?ȸ
Ƞ
ndcg?prec??

_mult???

_avgjg???
?
inG?pref???
??
bpref?map???
??
nfAP?prec_at_recall??
ؐ

_???
avgjg?cut???
Ȟ

_???
cut?rel???
dcg?um_???
nonrel_judged_ret?q?	re?	ؒ
??

_ret?	Ȕ
l?	t?	??
ذ

_?	??

_ret?	?
Rnonrel?	imp?	ret?	ص
??
avgjg?	num_prefs_?
pair?
simp?
??

_ret?
?
ful?
poss?
ȭ
??

_?
??
imp?
ret?
Ȳ
??

_?
??
imp?
ret???
??
all?ip_rank???
c?l???
ative_P?string??
e?unid???
??
F?P?map?re?ȩ
??
??
call?lative_P??
et_?uccess?Ф
??
?
Ȇ
form_inter_procs?re?
trec_measure?
??
l_info_format?sults_format?
??
А
_nicknames?
s?
??
??
a_cut?
empty?
s_?
?
??

_p?
??
float?
long???
final_meas_?
single_meas_???
a_cut?empty?s_???
؂
float?long???
??
l_info_format?sults_format???
??
_nicknames?s??
?$??
??
????
$?
?
???
???
???
??,p X?
 ?
?
??
?
??
??????
p?
?	?
p??	?
p???
p???
?
??
?
?
4???
?
4???
???
??
?
?	?
??
??
44,?
????
?
?
?
??
p??
?
??@l?
?
??
????????????
,?
?
??
,?
??
,????
,?????4p?l?
?
?
?
?
??
???
??????????
??????K

d
Y

!
p

?!
?

0$
?

,&
?

0'
?

?+
?

L.
?

?0
?

`3
?

?5

?7

T:
#
=
.
H=
:
 N
L
?R
b
?Y
v
_
?
Xe
?
?j
?
?p
?
|u
?
@}
?
?


̍

??
/
,?
;
Ԝ
M
??
]
4?
i
8?

 ?
?
д
?
??
?
L?
?
t?
?
??
?
d?
?
??


?
%

??
@

|?
]

??
m

?
y

P?
?

??
?

??
?

??
?

l?
?

??

??
"
??
5
$?
J
??
T
??
_
??
l
??
w
??
?
??
?
,?
?
??
?
D?
?
??
?
`?
?
??

?

?
'
??
@
??
P
??
d
??
u
,

?
?

?
4	

?
x

?



?
H


?
?

?
<

?
?

?
$



P


 $

 
d'

*
(

9
4(

G
?)

R
?/

_
?2

i
?3

x
?3

?
6

?
?7

?
$:

?
 <

?
,?

?
`?

?
??

?
?A


HB

&
?B

5
?C

I
@D

a
(E

r
?E

?
G

?
?G

?
?H

?
?I

?
?J


HL

$
?O

9
dQ

V
?S

c
,\

{
P]

?
_

?
P`

?
xb

?
i

?
dj



(m

#
?n

6
xp

P
?t

a
\w

o?
??
??
??
?P?
?X?
?`?
??
?x?
?؅
???
?x?
??
??
1?
D؋
e@?
y??
??
?H?
???
??
?h?
???
??
)??
??
R??
n?
?8?
???
??
?p?
?X?
???

?
p?
*x?
G??
[0?
t??
?0?
?8?
?@?
?H?
?P?
?X?
?`?
?h?
p?
x?
+??
;??
L??
\??
k??
{??
???
???
???
?ȝ
?Н
?؝
??
 ?
(?
&0?
28?
B@?
NH?
^P?
gX?
l`?
h?
?p?
?x?
???
???
???
???
???
???
???
??
??
Ȟ
*О
=؞
N??
c?
l?
y??
??
??
??
??
? ?
?(?
?0?
?8?
?@?
?H?

P?
X?
,`?
>h?
Lp?
bx?
t??
???
???
???
???
???
?П
??
?

?
?

d
)dKdWf
9??e
.

?$


$h
N


.
d
?$
d

$?

N
d

.
!
?$
!

$?
N
!

.
?!
?$
?!

$d
N
?!

.
0$
?$
0$

$?

N
0$

.
,&
?$
,&

$

N
,&

.
0'
?$
0'

$?
N
0'
&?
&?
%&?
,&?

d
:d\dff
9??e? ? ? ? ?  
d
,dNdZf
9??e
.
?'
?$
?'

$$
N
?'

.
 (
?$
 (

$?
N
 (

.
?(
?$
?(

$?
N
?(

.
?)
?$
?)

$
N
?)

.
?+
?$
?+

$?
N
?+

.
L.
$
L.

$?
N
L.

.
?.
$
?.

$
N
?.

.
?0
;$
?0

$l
N
?0

.
`3
N$
`3

$?
N
`3

.
?3
\$
?3

$?

N
?3

.
?5
z$
?5

$?

N
?5

.
?7
?$
?7

$?
N
?7

.
l8
?$
l8

$?

N
l8

.
T:
?$
T:

$?
N
T:

.
=
?$
=

$,
N
=

.
H=
?$
H=

$p
N
H=

d
?dd
f
9??e
.
?=
P$
?=

$ 
N
?=

.
?=
c$
?=

$X
N
?=

.
0>
r$
0>

$?
N
0>

d
?d?d?f
9??e
.
?>
?$
?>

$ 
N
?>

.
?>

 $
?>

$?
N
?>

.
t?
! $
t?

$?
N
t?

.
8@
9 $
8@

$

N
8@

d
P dr d? f
9??e
.
/d`/dn/f
9??e
.
t?
?/$
t?

$0
N
t?

.
??
?/$
??

$?
N
??

.
??
?/$
??

$?
N
??

.
$?
?/$
$?

$p
N
$?
0&??
0&??
$0&??

d
70dY0dk0f
9??e
.
??
?0$
??

$?

N
??

.
??
?0$
??

$?
N
??

.
??
?0$
??

$?
N
??

d
?0d1d
1f
9??eP1 b1 x1&?
?1&x?
?1&؅
?1&??
?1&x?
?1&??
?1 ?1 
d

2d#2d+2f
9??e
.
??
k2$
??

$4

N
??
x2 
d
?2d?2d?2f
9??e
.
??
?2$
??

$?

N
??
?2&??
	3 3&?

d
'3dI3dS3f
9??e
.
??
?3$
??

$@
N
??

.
??
?3$
??

$l
N
??
?3 
d
?3d?3d?3f
9??e
.
,?
84$
,?

$?
N
,?
I4 
d
Z4d|4d?4f
9??e
.
??
?4$
??

$?
N
??

.
D?
?4$
D?

$?
N
D?
?4 
d
5d%5d55f
9??e
.
??
}5$
??

$?
N
??
?5 
d
?5d?5d?5f
9??e
.
`?
6$
`?

$t

N
`?
'6 
d
76dY6dc6f
9??e
.
??
?6$
??

$D

N
??
?6 
d
?6d?6d?6f
9??e
.
?
;7$
?

$

N
?
O7 
d
c7d?7d?7f
:??e
.
?
?7$
?

$?
N
?
?7 
d
?7d8d%8f
:??e
.
??
q8$
??

$0
N
??
?8&؋
?8 ?8&@?

d
?8d?8d9f
:??e
.
??
H9$
??

$?

N
??
X9&??
p9 ?9&?

d
?9d?9d?9f
:??e
.
??
:$
??

$(
N
??
:&H?
6: J:&??

d
^:d?:d?:f
:??e
.
??
?:$
??

$X

N
??
?:&?
?: 
;&h?

d
;dA;dN;f
:??e
.
,

?;$
,


$x
N
,

?;&??
?; ?;&??

d
?;d<d<f
:??e
.
?

O<$
?


$?
N
?


.
4	

]<$
4	


$D
N
4	


.
x

j<$
x


$?
N
x


.



t<$




$,
N



?< ?<&??

d
?<d?<d?<f
:??e
.
H


=$
H



$D
N
H


+=&??
A= S=&?

d
f=d?=d?=f
:??e
.
?

?=$
?


$?
N
?


.
<

?=$
<


$D
N
<


.
?

?=$
?


$?
N
?


.
$

?=$
$


$,
N
$

	> >&??

d
,>dN>d[>f
:??e
.
P

?>$
P


$?
N
P


.
 $

?>$
 $


$D
N
 $


.
d'

?>$
d'


$?
N
d'


.
(

?>$
(


$,
N
(

?> ?>&П

d
?>d ?d)?f
:??e
.
4(

j?$
4(


$?

N
4(

x? 
d
??d??d??f
:??e
.
?)

??$
?)


$?
N
?)


.
?/

??$
?/


$D
N
?/


.
?2

@$
?2


$?
N
?2


.
?3

@$
?3


$,
N
?3

@ (@&?

d
9@d[@de@f
:??e
.
?3

?@$
?3


$P
N
?3

?@&??
?@ ?@&?

d
?@dAd'Af
:??e
.
6

kA$
6


$?

N
6

|A&8?
?A ?A&??

d
?Ad?Ad?Af
:??e
.
?7

*B$
?7


$P
N
?7

9B 
d
HBdjBdvBf
:??e
.
$:

?B$
$:


$?

N
$:

?B&?
?B ?B&p?

d
Cd&Cd3Cf
:??e
.
 <

xC$
 <


$
N
 <

?C 
d
?Cd?Cd?Cf
:??e
.
,?


D$
,?


$4
N
,?


.
`?

D$
`?


$p
N
`?

)D 8D&?

d
?DdaDdoDf
:??e
.
??

?D$
??


$
N
??


.
?A

?D$
?A


$l
N
?A


.
HB

?D$
HB


$?
N
HB

?D&X?
E E&??
.E&?
>E&?

d
JEdlEdvEf
:??e
.
?B

?E$
?B


$?
N
?B

?E 
d
?Ed?EdFf
:??e
.
?C

NF$
?C


$?
N
?C

bF 
d
vFd?Fd?Ff
:??e
.
@D

?F$
@D


$?
N
@D

G 
d
Gd>GdJGf
:??e
.
(E

?G$
(E


$?
N
(E

?G 
d
?Gd?Gd?Gf
:??e
.
?E

H$
?E


$

N
?E

-H&?
CH RH&p?

d
eHd?Hd?Hf
:??e
.
G

?H$
G


$?
N
G

I 
d
1IdSIdlIf
:??e
.
?G

?I$
?G


$$

N
?G

?I 
d
?IdJd3Jf
:??e
.
?H

?J$
?H


$

N
?H

?J 
d
?Jd?Jd?Jf
:??e
.
?I

OK$
?I


$?
N
?I

pK 
d
?Kd?Kd?Kf
:??e
.
?J

	L$
?J


$?

N
?J

L 
d
1LdSLdbLf
:??e
.
HL

?L$
HL


$x
N
HL

?L 
d
?Ld?LdMf
:??e
.
?O

KM$
?O


$?

N
?O

`M 
d
uMd?Md?Mf
:??e
.
dQ

?M$
dQ


$?
N
dQ


.
?S

N$
?S


$@
N
?S

)N 
d
FNdhNd{Nf
:??e
.
,\

?N$
,\


$$

N
,\

?N 
d
?NdOd+Of
:??e
.
P]

vO$
P]


$?

N
P]

?O 
d
?Od?Od?Of
:??e
.
_

(P$
_


$D

N
_

AP 
d
ZPd|Pd?Pf
:??e
.
P`

?P$
P`


$(
N
P`


.
xb


Q$
xb


$?
N
xb

Q 
d
;Qd]QdpQf
:??e
.
i

?Q$
i


$d

N
i

?Q 
d
?Qd
Rd Rf
:??e
.
dj

kR$
dj


$?
N
dj

?R 
d
?Rd?Rd?Rf
:??e
.
(m

S$
(m


$?

N
(m

6S 
d
OSdqSdSf
;??e
.
?n

?S$
?n


$?

N
?n

?S 
d
?Sd
Td"Tf
;??e
.
xp

oT$
xp


$D
N
xp

?T&x?
?T ?T&??

d
?Td?TdUf
;??e
.
?t

FU$
?t


$?
N
?t

WU&0?
pU ?U&??

d
?Ud?Ud?Uf
;??e
.
\w

V$
\w


$<

N
\w

V 
d



h?
)

/
0>
B
?=
U
?=
d
t?
|
?>
?
?>
?
8@
?
??
?
??
?
??
?Ȃ


?
.

??
@

?
Z

`?
o

??
?

?_
?

?i
?

?H
?

?Q
?

?u
?

 ?
?

4k

?t
.
(S
C
D^
`
t?
p
??
?
?.
?
?)
?
?'
?
 (
?
?3

l8
,
?(
A??
S8?
^??
iH?
zH?
??
?`?
???
??
???
???
???
??
??
-X?
:(?
M ?
^H?
l??
~??
?x?
?X?
???
?H?
???
?X?

??
'?
Hؚ
a??
z?
?h?
?Ȗ
??
???
H?
??
.8?
F??
^??
n8?
???
?p?
??
? ?
?Ȕ
???
??
?h?
P?
$?
5??
CH?
\??
t??
?P?
??
?
??tF??j?????͢????5??C{?6J???ik?ۖ?_h`?q?(*{(xj?	????p??V?_O?k
T?qd?r?b?E?ێ?BS>?^??6JD?oO}c?7C{T??K@1?8?GE?h?w??`?55?!lL??????`l??8C8??*???Q??4???ޔ?m??锞??;?.??$q????Tj[??2?MQ_?}bŔ
??$??
u?_????R]d?? ?R?+??1??_%?yф%|???@@??F?.?cԮ?Ro?k
Rh3??K?+X?[??8???1??&V??x???͜_J?9??䇵I{v?9??e?ݛ????;3?vn?o???df	gR?\???S????,3[ʗ?^4Ε+???y|???I?\So?N?2#??o??Wb???UNh??aD???u???????[???z?q??lʚbW?Z??:f?f)?G??)???զ?G?3JrE?@????1?U???=?p??vr?|`???h??[?"?/??D?&5?.?3g??=?6)?	???Z.?lV?N?d??8???[??{?}?I???4(b?=?~ق???Ԍ?p???-W/??鶕?S????E??????Dk???????q%?ow??$!?
5p???)?f؃{?u?[NX8;?ן1?Y?????*??R?R?}?	??H+??0??c1`???3??;}|??Q?^???!cUf???~??o?s?]^b!?8???̆?;[=HD?	
1?H?m?a?5?\?sӂ?0?M~?S???kD7?P܆????N?ߡ֗?f??0??????|݉??L??a???T+?4:9??r?]g?YOB?ۿ?,?.
fs~\a??b?????
?
?v??!??J??
ӳ裟??ƒ???7???˿?]?.?h?!dhSeJ%^܀?Ջ-?Z??ɉ?M?!?Ӽ??'M?N(?<5!c??O0??|?o"JU?g??v
????i?)%???k_?;?I?}?\;???Xo??f????kOX?|?|z?ڽ?H?,????Xo??f????kOX?|?|z?ڽ?H?,????Xo??f????kOX?|?|z?ڽ?H?,?u#??Un?B???*????A?Yv?D??H??I:?[/??T?(????J???g??ѧ\?K?ġ?o???Xo??f????kOX?|?|z?ڽ?H?,????Xo??f????kOX?|?|z?ڽ?H?,?c???z7?o?@ydr?~e6??/??7&!fj??
?|?&`?H?i?+l?S??[r?#????6{D??b?݃7+?9?kX?Y?qY+%???n????X??c??
?$ɏ?2nP41_?IORy?E???w?S???ˎH??????U?̋??=???~Ҵ??_}"?r'b?
?ZĊ??Ὼ????x4
?S??k??3B?`
????
??J??2???)%?=?^?r=/