CAFASP3 SCOP assessment

Results based on SCOP 1.61 (and Alexey Murzin's provisional classification of CASP targets)

Furthermore this is a goal-driven assessment. It provides an objective ranking for the servers that aim at the prediction of protein fold and probable evolutionary relationship as classified in SCOP. Note that many servers do not necessarily have the same goal but instead aim at the recognition of a large fragment of similar structure in a protein of known structure regardless of its SCOP classification. I think that for such servers this ranking will be largerly irrelevant.

Also available is the same evaluation of LiveBench results.

SCOP assessment of templates for the 56 targets in CAFASP 3.

Rank	Included	Server	Sensitivity	Specifity	1	2	3	4	5	6	7	8	9	10
1	56	fugu3	41	40.7	38	41	41	41	41	41	41	41	41	41
1	55	ffas03	41	40.5	37	40	41	41	41	41	41	41	41	41
1	56	fugsa	41	40.2	38	40	40	40	40	40	41	41	41	41
1	55	pmodel3	41	40.1	36	40	40	40	40	41	41	41	41	41
1	56	3ds5	41	40.1	38	38	38	41	41	41	41	41	41	41
1	56	3dsn	40	40.0	40	40	40	40	40	40	40	40	40	40
1	56	3ds3	40	40.0	40	40	40	40	40	40	40	40	40	40
2	53	pcons3	40	39.1	36	37	39	39	40	40	40	40	40	40
2	55	pmodel	40	39.0	39	39	39	39	39	39	39	39	39	39
2	55	inbgu	40	38.6	38	38	38	38	38	39	39	39	39	40
2	55	shgu	39	38.6	37	37	39	39	39	39	39	39	39	39
2	54	pcons2	39	38.5	36	37	39	39	39	39	39	39	39	39
3	56	supfampp	38	37.0	37	37	37	37	37	37	37	37	37	37
3	55	samt99	39	36.9	34	36	37	37	37	37	37	37	38	39
3	56	mgenthreader	39	36.3	18	37	38	38	38	38	39	39	39	39
3	55	forte1	39	36.2	34	34	35	35	35	35	37	39	39	39
3	53	orfeus	38	36.0	32	33	34	37	37	37	37	37	38	38
3	54	foldfit	37	35.8	32	33	34	37	37	37	37	37	37	37
4	56	superfamily	35	35.0	35	35	35	35	35	35	35	35	35	35
4	56	mpalign	36	35.0	35	35	35	35	35	35	35	35	35	35
5	55	burnham	35	34.3	31	33	34	35	35	35	35	35	35	35
5	56	genthreader	36	34.0	30	34	34	34	34	34	34	34	36	36
5	56	pcomb	36	33.5	26	28	29	36	36	36	36	36	36	36
5	34	alax	33	33.0	33
5	52	pspt	36	32.8	31	33	33	33	33	33	33	33	33	33
5	56	raptor	37	32.3	27	28	32	32	34	34	34	34	34	34
5	53	pdbblast	32	32.0	32	32	32	32	32	32	32	32	32	32
5	56	prospect	38	31.4	14	30	30	31	31	34	34	36	37	37
6	42	samt02	30	29.7	27	30	30	30	30	30	30	30	30	30
7	45	jigsaw	33	27.4	23	25	25	27	28	28	28	29	29	32
8	27	pilot	21	21.0	21	21	21	21	21	21
9	38	libellula	26	19.3	3	13	17	18	20	24	24	24	24	26
9	52	rpfold	19	19.0	19	19	19	19	19	19	19	19	19	19
10	47	protinfocm	23	17.6	11	15	18	18	18	18	18	20	20	20
11	43	loopp	12	8.2	6	7	7	8	9	9	9	9	9	9
12	13	frost	5	4.9	4	5	5	5	5	5	5	5
13	25	cnbpred	4	1.8	0	2	2	2	2	2	2	2	2	2
14	44	protfinder	7	0.7	0	0	0	1	1	1	1	1	1	1

The sensitivity score is the total number of true positives. The specificity score is the average of the 10 columns. Each column shows the number of true positivies before the 'n'th false positive, where 'n' is the column number. The first column shows the number of targets which template information was available for.

The full data is available in a flat file here. Also alternative tables which classify at the fold or superfamily level, and can include the targets split into domains.

General procedure

A single template is assumed for every model.

The template used for each model will be the first one listed in the 'PARENT' field of the submitted model (raw submission).

All SCOP domains in the template and the target will be compared.

If any pair of domains in the target and the template belong to the same SCOP superfamily, the model is judged as 'true'.

When judging at the superfamily level, if any pair of domains in the target and the template belong to the same SCOP fold, but no pair belongs to the same superfamily, the model is judged as ambiguous and is neither 'true' nor 'false'.

If no pair of domains in the target and the template belong to the same SCOP fold, the model is judged as 'false'.

There are documented exceptions to the classification in SCOP, and these are taken into consideration wherever humanly possible. The major ones are listed in the following three points in the list.

If any pair of domains in the target and the template belong to the TIM-barrel fold the model is judged as 'true'.

If any pair of domains in the target and the template belong to any of the Rossmann folds (NAD(P), FAD/NAD(P), or Nucleotide binding domains) the model is judged as 'true'.

The families in the Membrane all-alpha superfamily are more like superfamilies and are treated accordingly. If the template and the target belong to different families within the Membrane all-alpha superfamily, the model is judged as 'false'.

Limitations

The classification of the targets into SCOP is provisional and generously provided with the caveat that it may change before the next SCOP release.

There is a small chance that an incorrect model will be judged 'true' when built from a template which by chance contains a SCOP domain in the same superfamily as the target, although the model does not use that part of the template. As this is unlikely to happen often by chance, it is not expected to affect the results.

Models which are built from more than one template will only have one template (the first listed) considered. Since a binary (true/false) decision is being made, by judging only the first listed template, the result is not affected unless there are other templates listed which would be 'true' when the first is 'false'.

Methods using models which make modifications to the backbone of the template will not be as relevant for this comparison.

It is not possible in all cases to extract a template at all. This is reflected in the numbers in the first column of the tables.

Templates using theoretical models are ignored. Templates which have no record in PDB are ignored. Templates which are old or have been replaced are kept and the classification obtained from old SCOP releases.

Results

Acknowledgements