LiveBench SCOP assessment

Results based on SCOP 1.61

Rank	Included	Server	Sensitivity	Specifity	1	2	3	4	5	6	7	8	9	10
1	106	dali	69	64.8	56	57	65	67	67	67	67	67	67	68
2	106	3dhit	68	55.9	51	53	53	55	55	56	56	58	61	61
3	99	3ds5	55	52.6	47	52	52	53	53	53	54	54	54	54
3	106	shgu	59	52.2	48	48	49	49	54	54	54	54	56	56
3	106	pcons2	55	52.0	52	52	52	52	52	52	52	52	52	52
3	106	fugu3	58	51.4	48	48	50	50	53	53	53	53	53	53
3	106	fugsa	58	51.4	48	48	50	50	53	53	53	53	53	53
3	97	3ds3	55	51.0	47	47	49	49	52	52	52	54	54	54
4	106	burnham	54	49.8	40	48	50	50	50	52	52	52	52	52
4	106	samt99	52	49.6	41	49	49	51	51	51	51	51	51	51
5	106	orfeus	57	48.5	27	48	50	50	51	51	52	52	52	52
5	106	inbgu	58	48.4	45	45	47	47	48	48	51	51	51	51
6	106	orfblast	51	47.2	27	47	48	48	48	50	51	51	51	51
7	106	fugue	49	46.1	42	42	43	44	46	48	49	49	49	49
8	106	superfamily	48	44.8	43	45	45	45	45	45	45	45	45	45
9	106	foldfit	59	43.0	36	36	37	43	43	46	47	47	47	48
9	106	mgenthreader	53	42.7	13	41	41	44	47	48	48	48	48	49
10	106	pdbblast	46	41.0	36	37	41	41	41	42	42	42	44	44
11	106	genthreader	50	33.4	9	25	35	35	37	37	37	38	40	41
12	104	blast	20	5.1	2	2	4	5	5	6	6	7	7	7

The sensitivity score is the total number of true positives. The specificity score is the average of the 10 columns. Each column shows the number of true positivies before the 'n'th false positive, where 'n' is the column number.
Of the 106 targets 79 had existing templates, 33 were completely novel. Of these 33 novel structures, 7 had a similar structure come out around the same time, but I have assumed that it was afterwards.
The full data is available in a flat file here.

Declaration

General procedure

Rules

A single template will be assumed for every model.

The template used for each model will be the one currently displayed on the LiveBench results page (however that is chosen).

All SCOP domains in the template and the target will be compared.

If any pair of domains in the target and the template belong to the same SCOP superfamily, the model is judged as 'true'.

If any pair of domains in the target and the template belong to the same SCOP fold, but no pair belongs to the same superfamily, the model is judged as ambiguous and is neither 'true' nor 'false'.

If no pair of domains in the target and the template belong to the same SCOP fold, the model is judged as 'false'.

Documented exceptions to the rules will be allowed where there is a note in SCOP indicating a true relationship above the superfamily level. The major ones are listed in the following three points in the list.

If any pair of domains in the target and the template belong to the TIM-barrel fold the model is judged as 'true'.

If any pair of domains in the target and the template belong to any of the Rossmann folds (NAD(P), FAD/NAD(P), or Nucleotide binding domains) the model is judged as 'true'.

The families in the Membrane all-alpha superfamily are actually superfamilies and are treated accordingly. If the template and the target belong to different families within the Membrane all-alpha superfamily, the model is judged as neither 'true' nor 'false'.

Limitations

There is a small chance that an incorrect model will be judged 'true' when built from a template which by chance contains a SCOP domain in the same superfamily as the target, although the model does not use that part of the template. As this is unlikely to happen often by chance, it is not expected to affect the results.

Models which are built from more than one template will only have one template considered. Since a binary (true/false) decision is being made, the judging of one template does not affect the decision unless there are other templates used which would be 'true' when that which is displayed is 'false'.

Methods using models which make modifications to the backbone of the template will not be as relevant for this comparison.

Feedback