Overall Results

Here we present the average and standard deviation results of the current benchmark version. We also point to the corresponding .yml configuration file used for each configuration so that users can consistently reproduce experiments or build new configurations based on one of them.

ml-100k

Experiment ran using the MovieLens-100k dataset with the following presented models and their configurations. The complete configuration can be found in config_files/run_ml-100k.yml and config_files/run_gnns.yml:

  • Summarized results from experiment_results/fixed_db16_runs/ml-100k.csv and experiment_results/fixed_db16_runs/ml-100k_gnns.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.0993 ± .0034

.1766 ± .0043

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.0973 ± .0039

.1748 ± .0064

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0032 ± .0003

.0077 ± .0004

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0009 ± .0003

.0023 ± .0004

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0029 ± .0003

.0070 ± .0006

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0047 ± .0003

.0113 ± .0005

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0031 ± .0004

.0074 ± .0005

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0051 ± .0003

.0120 ± .0007

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0045 ± .0006

.0109 ± .0013

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0048 ± .0010

.0113 ± .0015

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0042 ± .0003

.0103 ± .0007

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=movie_title;iterations=30;mi=0.5

.0017 ± .0003

.0039 ± .0005

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.0985 ± .0041

.1761 ± .0058

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

.0069 ± .0004

.0158 ± .0006

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.0161 ± .0015

.0375 ± .0034

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0174 ± .0011

.0393 ± .0021

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0162 ± .0015

.0376 ± .0031

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0168 ± .0014

.0387 ± .0031

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-100k_times.csv and experiment_results/fixed_db16_runs/ml-100k_gnns_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

8.178 ± .1823

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

7.370 ± .6110

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

40.99 ± .3187

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

66.84 ± 1.996

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

79.29 ± 2.058

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

75.79 ± 2.347

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

382.2 ± 2.905

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

57.97 ± 2.069

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

39.87 ± 1.691

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

44.04 ± 1.478

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

47.50 ± 1.706

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=movie_title;iterations=30;mi=0.5

66.40 ± .0470

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

50.83 ± .6383

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

74104 ± 3749.

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

2774. ± 460.9

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

37254 ± 1163.

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

3242. ± 266.1

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

38941 ± 1146.

ml-100k_enriched

Experiment ran using the MovieLens-100k dataset with DBpedia enrichement and the following presented models and their configurations. The complete configuration can be found in config_files/run_ml-100k_enriched.yml and config_files/run_gnns.yml:

  • Summarized results from experiment_results/fixed_db16_runs/ml-100k_enriched.csv and experiment_results/fixed_db16_runs/ml-100k_enriched_gnns.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.1439 ± .0016

.2349 ± .0019

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.1433 ± .0033

.2329 ± .0037

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0033 ± .0002

.0082 ± .0003

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0007 ± .0001

.0018 ± .0002

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0022 ± .0004

.0058 ± .0009

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0044 ± .0003

.0106 ± .0003

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0029 ± .0005

.0070 ± .0009

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0050 ± .0004

.0122 ± .0008

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0033 ± .0008

.0081 ± .0018

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0046 ± .0005

.0112 ± .0008

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0045 ± .0003

.0108 ± .0008

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.0046 ± .0003

.0107 ± .0005

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.1442 ± .0022

.2350 ± .0019

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

.0056 ± .0013

.0136 ± .0028

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.2887 ± .0036

.3852 ± .0049

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0410 ± .0011

.0834 ± .0017

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.2915 ± .0048

.3880 ± .0068

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.2740 ± .0030

.3707 ± .0034

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-100k_enriched_times.csv and experiment_results/fixed_db16_runs/ml-100k_enriched_gnns_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

21.69 ± .2804

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

21.95 ± .6008

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

42.86 ± .8906

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

71.26 ± 1.692

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

113.8 ± 1.304

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

80.22 ± .9842

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

384.1 ± 2.356

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

62.31 ± .6928

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

52.90 ± 1.164

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

46.75 ± 1.562

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

86.69 ± 1.153

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

108.0 ± 1.952

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

85.93 ± .7969

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

75651 ± 2808.

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

7498. ± 1095.

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

54686 ± 1834.

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

9050. ± 527.7

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

52116 ± 4862.

ml-1m

Experiment ran using the MovieLens-1m dataset with the following presented models and their configurations. The complete configuration can be found in config_files/run_ml-1m.yml and config_files/run_gnns.yml:

  • Summarized results from experiment_results/fixed_db16_runs/ml-1m.csv, experiment_results/fixed_db16_runs/ml-1m_bPRMF.csv, experiment_results/fixed_db16_runs/ml-1m_cFKG.csv, experiment_results/fixed_db16_runs/ml-1m_cKE.csv and experiment_results/fixed_db16_runs/ml-1m_kGAT.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.0846 ± .0017

.1449 ± .0024

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.0846 ± .0010

.1454 ± .0011

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0026 ± .0001

.0063 ± .0003

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0009 ± .0001

.0021 ± .0001

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0007 ± .0001

.0016 ± .0002

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0014 ± .0001

.0036 ± .0003

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0003 ± .0001

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0050 ± .0001

.0115 ± .0001

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0010 ± .0002

.0025 ± .0005

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0012 ± .0004

.0030 ± .0010

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0025 ± .0001

.0062 ± .0004

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=movie_title;iterations=30;mi=0.5

.0028 ± .0002

.0062 ± .0004

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.0843 ± .0011

.1445 ± .0017

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.0794 ± .0013

.1075 ± .0018

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0310 ± .0005

.0521 ± .0006

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0798 ± .0008

.1077 ± .0011

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0646 ± .0005

.0902 ± .0004

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-1m_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

31.10 ± .2524

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

33.11 ± 2.733

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

423.6 ± 17.24

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

715.0 ± 15.18

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

864.0 ± 16.52

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

858.0 ± 16.76

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

3888. ± 29.29

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

588.1 ± 22.05

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

398.2 ± 19.76

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

472.00 ± 20.89

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

693.2 ± 21.37

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=movie_title;iterations=30;mi=0.5

499.3 ± 8.566

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

451.9 ± 6.538

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-1m_bPRMF_times.csv, experiment_results/fixed_db16_runs/ml-1m_cFKG_times.csv, experiment_results/fixed_db16_runs/ml-1m_cKE_times.csv and fixed_db16_runs/ml-1m_kGAT_times.csv (configuration: CPU: Apple M3 Ultra; RAM: 256GB; GPUs: []):

|BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]|13308 ± 287.8| |CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi|44416 ± 1019.| |CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001|22654 ± 500.8| |KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi|111312 ± 2031.|

ml-1m_enriched

Experiment ran using the MovieLens-1m dataset with DBpedia enrichement and the following presented models and their configurations. The complete configuration can be found in config_files/run_ml-1m_enriched.yml:

  • Summarized results from experiment_results/fixed_db16_runs/ml-1m_enriched.csv, experiment_results/fixed_db16_runs/ml-1m_enriched_bPRMF.csv, experiment_results/fixed_db16_runs/ml-1m_enriched_cFKG.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.1254 ± .0018

.1961 ± .0026

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.1253 ± .0042

.1957 ± .0048

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0027 ± .0002

.0065 ± .0005

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0010 ± .0001

.0023 ± .0001

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0005 ± .0001

.0013 ± .0002

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0015 ± .0001

.0037 ± .0002

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0004 ± .0001

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0051 ± .0001

.0116 ± .0003

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0027 ± .0002

.0066 ± .0005

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0009 ± .0002

.0024 ± .0006

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0024 ± .0001

.0066 ± .0005

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.0019 ± .0003

.0044 ± .0004

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.1252 ± .0017

.1964 ± .0020

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.0803 ± .0011

.1079 ± .0011

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-1m_enriched_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

62.50 ± 3.393

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

65.73 ± 5.698

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

428.7 ± 8.749

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

718.4 ± 7.618

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

1114. ± 14.14

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

855.1 ± 13.96

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

3907. ± 15.04

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

594.5 ± 16.06

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

448.9 ± 13.98

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

476.3 ± 14.38

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

1127. ± 15.99

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

592.6 ± 5.611

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

520.1 ± 4.842

  • Summarized execution time results from experiment_results/fixed_db16_runs/ml-1m_enriched_bPRMF_times.csv, experiment_results/fixed_db16_runs/ml-1m_enriched_cFKG_times.csv, experiment_results/fixed_db16_runs/ml-1m_enriched_cKE_times.csv and experiment_results/fixed_db16_runs/ml-1m_enriched_kGAT_times.csv (configuration: CPU: Apple M3 Ultra; RAM: 256GB; GPUs: []):

Model

Execution Time (s)

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

13913 ± 230.0

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

43210 ± 772.5

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

24348 ± 416.1

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

149675 ± 6930

lastfm

Experiment ran using the Lastfm dataset with the following presented models and their configurations. The complete configuration can be found in config_files/run_lastfm.yml and config_files/run_gnns.yml:

  • Summarized results from experiment_results/fixed_db16_runs/lastfm.csv and experiment_results/fixed_db16_runs/lastfm_gnns.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.0741 ± .0017

.1753 ± .0042

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.0756 ± .0015

.1782 ± .0047

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0001 ± .0000

.0002 ± .0001

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0005 ± .0001

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0001 ± .0000

.0002 ± .0001

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0002 ± .0000

.0005 ± .0002

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0004 ± .0001

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0001 ± .0000

.0004 ± .0003

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0001 ± .0000

.0003 ± .0002

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0002 ± .0001

.0004 ± .0003

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0002 ± .0000

.0005 ± .0002

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=name;iterations=30;mi=0.5

.0002 ± .0000

.0005 ± .0002

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.0757 ± .0025

.1774 ± .0047

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.1032 ± .0013

.2362 ± .0036

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0031 ± .0026

.0094 ± .0073

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.1049 ± .0033

.2380 ± .0085

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.1017 ± .0021

.2342 ± .0032

  • Summarized execution time results from experiment_results/fixed_db16_runs/lastfm_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

66.77 ± 1.416

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

64.84 ± 3.724

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

56.40 ± .6367

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

87.21 ± 1.854

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

449.6 ± 2.516

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

111.0 ± 1.613

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

381.7 ± 2.037

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

104.8 ± 2.123

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

53.39 ± 1.559

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

63.46 ± 1.600

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

153.0 ± 2.265

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=name;iterations=30;mi=0.5

262.9 ± .5831

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

144.7 ± 1.818

  • Summarized execution time results from experiment_results/fixed_db16_runs/lastfm_gnns_times.csv (configuration: CPU: Apple M3 Ultra; RAM: 256GB; GPUs: []):

Model

Execution Time (s)

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

1883. ± 72.75

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

7920. ± 67.19

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

2537. ± 373.2

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

19364 ± 1473.

lastfm_enriched

Experiment ran using the Lastfm dataset with DBpedia enrichement and the following presented models and their configurations. The complete configuration can be found in config_files/run_lastfm-enriched.yml and config_files/run_gnns.yml:

  • Summarized results from experiment_results/fixed_db16_runs/lastfm_enriched.csv and experiment_results/fixed_db16_runs/lastfm_enriched_gnns.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.0840 ± .0021

.1988 ± .0036

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.0839 ± .0021

.2000 ± .0040

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0001 ± .0001

.0004 ± .0002

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0004 ± .0003

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0001 ± .0000

.0001 ± .0000

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0001 ± .0001

.0002 ± .0002

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0001 ± .0000

.0003 ± .0001

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0002 ± .0001

.0004 ± .0001

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0002 ± .0001

.0005 ± .0002

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0002 ± .0000

.0004 ± .0001

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0002 ± .0000

.0004 ± .0002

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.0001 ± .0000

.0002 ± .0001

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.0840 ± .0010

.1992 ± .0022

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.1035 ± .0027

.2392 ± .0035

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.0015 ± .0011

.0045 ± .0030

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.1035 ± .0030

.2394 ± .0043

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.1005 ± .0043

.2327 ± .0099

  • Summarized execution time results from experiment_results/fixed_db16_runs/lastfm_enriched_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

171.7 ± 4.597

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

169.2 ± 2.115

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

69.51 ± 3.929

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

97.41 ± 3.335

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

1355. ± 4.618

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

122.5 ± 2.912

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

392.8 ± 3.085

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

116.9 ± 4.735

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

164.4 ± 6.892

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

75.28 ± 4.570

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

748.9 ± 3.893

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-roberta-large-v1;embed_with=abstract;iterations=30;mi=0.5

450.5 ± 1.035

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

368.2 ± 6.358

  • Summarized execution time results from experiment_results/fixed_db16_runs/lastfm_enriched_gnns_times.csv (configuration: CPU: Apple M3 Ultra; RAM: 256GB; GPUs: []):

Model

Execution Time (s)

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

6039. ± 103.1

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

22985 ± 317.9

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

8218. ± 92.23

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

57487 ± 8615.

douban-movie

Experiment ran using the Douban Movie dataset with the following presented models and their configurations. The complete configuration can be found in config_files/run_douban-movie.yml and config_files/run_gnns_douban-movie.yml:

  • Summarized results from experiment_results/fixed_db16_runs/douban-movie.csv and experiment_results/fixed_db16_runs/douban-movie_gnns.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.7416 ± .0080

.8109 ± .0065

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.7418 ± .0069

.8119 ± .0060

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.5839 ± .0049

.6737 ± .0032

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.5773 ± .0030

.6663 ± .0016

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.5900 ± .0041

.6784 ± .0021

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.5909 ± .0038

.6798 ± .0022

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.5927 ± .0052

.6800 ± .0027

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.5929 ± .0058

.6809 ± .0035

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.5915 ± .0033

.6781 ± .0014

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.5877 ± .0028

.6776 ± .0022

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.5911 ± .0036

.6792 ± .0022

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=name_EN;iterations=30;mi=0.5

.5925 ± .0035

.6586 ± .0018

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.7401 ± .0046

.8086 ± .0042

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

.5956 ± .0134

.6846 ± .0089

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.3091 ± .0036

.3452 ± .0029

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.3031 ± .0048

.3398 ± .0036

  • Summarized execution time results from experiment_results/fixed_db16_runs/douban-movie_times.csv and experiment_results/fixed_db16_runs/douban-movie_gnns_times.csv (configuration: CPU: AMD EPYC 7502P 32-Core Processor; RAM: 94GB; GPUs: [‘NVIDIA A2’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

834.3 ± 24.82

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

823.1 ± 29.22

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

110.7 ± 1.628

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

126.1 ± 3.252

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

140.5 ± 4.679

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

134.4 ± 5.008

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

404.2 ± 8.478

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

117.1 ± 4.573

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

103.4 ± 3.690

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

105.7 ± 4.065

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

117.3 ± 3.828

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=name_EN;iterations=30;mi=0.5

1099. ± 2.803

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

1488. ± 13.74

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

6276. ± 160.6

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

3369. ± 697.7

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

25624 ± 2786.

douban-movie_enriched

Experiment ran using the Douban Movie dataset with DBpedia enrichement and the following presented models and their configurations. The complete configuration can be found in config_files/run_douban-movie_enriched.yml:

  • Summarized results from experiment_results/fixed_db16_runs/douban-movie_enriched.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.7371 ± .0064

.8043 ± .0060

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.7367 ± .0050

.8039 ± .0048

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.5838 ± .0046

.6738 ± .0038

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.5777 ± .0029

.6669 ± .0025

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.5891 ± .0041

.6780 ± .0022

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.5913 ± .0032

.6799 ± .0015

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.5930 ± .0036

.6808 ± .0015

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.5935 ± .0049

.6819 ± .0030

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.5917 ± .0049

.6783 ± .0019

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.5874 ± .0042

.6774 ± .0020

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.5912 ± .0044

.6796 ± .0019

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.5283 ± .0032

.6225 ± .0019

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.7372 ± .0044

.8044 ± .0047

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

.5944 ± .0089

.6829 ± .0081

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.3136 ± .0035

.3544 ± .0020

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.3096 ± .0020

.3509 ± .0012

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.3135 ± .0036

.3543 ± .0021

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

.3116 ± .0025

.3527 ± .0015

  • Summarized execution time results from experiment_results/fixed_db16_runs/douban-movie_times_enriched.csv (configuration: CPU: AMD Ryzen 5 7600 6-Core Processor; RAM: 31GB; GPUs: [‘NVIDIA GeForce RTX 4060’]):

Model

Execution Time (s)

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

832.9 ± 15.72

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

856.7 ± 12.74

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

112.4 ± 4.230

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

129.6 ± 5.001

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

144.0 ± 5.130

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

138.6 ± 5.678

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

406.2 ± 9.683

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

120.4 ± 5.355

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

106.0 ± 4.017

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

108.0 ± 4.924

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

120.9 ± 5.727

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

1114. ± 7.497

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

1470. ± 16.06

Entity2Rec;embedding_model=deepwalk_based;embedding_model_kwargs={‘config’: {‘save_weights’: True}, ‘parameters’: {‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1}};run_all=False;workers=6;iterations=1;collab_only=False;content_only=False

6278. ± 167.4

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

3609. ± 736.6

CFKG;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

30844 ± 3791.

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

4083. ± 119.5

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi

45945 ± 7669.

mind-small

Experiment ran using the MIND-small dataset and the following presented models with their configurations. The complete configuration can be found in config_files/run_mind-small.yml and config_files/run_mind-small_gnns.yml:

  • Summarized results from the files experiment_results/mind/mind-small_*.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.1690 ± .0016

.0316 ± .0008

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.1694 ± .0024

.0318 ± .0007

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0000 ± .0000

.0000 ± .0000

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0061 ± .0022

.0026 ± .0011

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0009 ± .0001

.0002 ± .0000

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0000 ± .0000

.0000 ± .0000

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0001 ± .0000

.0000 ± .0000

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0001 ± .0000

.0000 ± .0000

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.0000 ± .0000

.0000 ± .0000

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.1688 ± .0030

.0317 ± .0009

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.0001 ± .0000

.0000 ± .0000

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0001 ± .0000

.0000 ± .0000

mind-small_enriched

Experiment ran using the MIND-small dataset and the following presented models with their configurations. The complete configuration can be found in config_files/run_mind-small.yml and config_files/run_mind-small_gnns.yml:

  • Summarized results from the files experiment_results/mind/mind-small_*.csv:

Model

MAP@10

nDCG@10

Node2Vec based model + cosine similarity;q=1.0;p=1.0;embedding_size=64

.1459 ± .0025

.0282 ± .0006

Node2Vec based model + cosine similarity;q=0.6;p=0.8;embedding_size=64

.1473 ± .0014

.0283 ± .0009

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;epochs=25;seed=42;triples=all

.0000 ± .0000

.0000 ± .0000

TransD based model + cosine similarity;embedding_dim=150;epochs=25;seed=42;triples=ratings

.0000 ± .0000

.0000 ± .0000

TuckER based model + cosine similarity;embedding_dim=200;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;epochs=25;seed=42;triples=ratings

.0001 ± .0001

.0000 ± .0000

RESCAL based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=ratings

.0007 ± .0000

.0000 ± .0000

DistMult based model + cosine similarity;embedding_dim=50;epochs=25;seed=42;triples=all

.0003 ± .0001

.0001 ± .0000

ComplEx based model + cosine similarity;embedding_dim=100;epochs=25;seed=42

.0001 ± .0000

.0000 ± .0000

RotatE based model + cosine similarity;embedding_dim=200;epochs=25;seed=42;triples=all

.0001 ± .0000

.0000 ± .0000

EPHEN based model + cosine similarity;embedding_model=sentence-transformers/all-mpnet-base-v2;embed_with=abstract;iterations=30;mi=0.5

.0006 ± .0012

.0002 ± .0004

EPHEN based model + cosine similarity;embedding_model=deepwalk_based;embedding_model_kwargs={‘walk_len’: 10, ‘p’: 1.0, ‘q’: 1.0, ‘n_walks’: 50, ‘embedding_size’: 64, ‘epochs’: 1};embed_with=graph;iterations=30;mi=0.5

.1447 ± .0039

.0278 ± .0009

BPRMF;embed_size=64;epoch=1000;regs[1e-05, 1e-05, 0.01]

.0000 ± .0000

.0000 ± .0000

CKE;epoch=1000;kge_size=64;embed_size=64;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0000 ± .0000

.0000 ± .0000

Embedding Dimension Ablation

ml-100k kge_embedding

Ablation experiment changin only the embedding_dim parameter of KGE models on the ml-100k dataset. The complete configuration for the first experiment with the recommended embedding dimension size can be found in config_files\kge_parameters\kge_embedding-1.yml, and the full results besides averages and standard deviations can be found in experiment_results\kge_parameters\ml-100k_kge_embedding-1.csv:

Model

MAP@10

nDCG@10

Precision@10

Recall@10

F-score@10

TransE based model + cosine similarity;embedding_dim=150;scoring_fct_norm=1;entity_initializer=None;relation_initializer=None;relation_constrainer=None;regularizer=None;epochs=25;seed=42;triples=all

.0032 ± .0003

.0078 ± .0006

.0097 ± .0006

.0050 ± .0005

.0066 ± .0006

TransH based model + cosine similarity;embedding_dim=150;scoring_fct_norm=2;entity_initializer=None;relation_initializer=None;relation_regularizer=None;epochs=25;seed=42;triples=all

.0046 ± .0005

.0110 ± .0013

.0137 ± .0007

.0064 ± .0007

.0088 ± .0008

TransR based model + cosine similarity;embedding_dim=150;relation_dim=90;scoring_fct_norm=2;entity_initializer=None;entity_constrainer=None;relation_initializer=None;relation_constrainer=None;epochs=25;seed=42;triples=all

.0013 ± .0002

.0030 ± .0006

.0041 ± .0005

.0021 ± .0005

.0028 ± .0005

TransD based model + cosine similarity;embedding_dim=150;relation_dim=None;entity_initializer=None;entity_constrainer=None;relation_initializer=None;relation_constrainer=None;epochs=25;seed=42;triples=all

.0043 ± .0005

.0103 ± .0009

.0124 ± .0013

.0065 ± .0005

.0085 ± .0007

TuckER based model + cosine similarity;embedding_dim=200;relation_dim=None;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;relation_initializer=None;core_tensor_initializer=None;epochs=25;seed=42;triples=all

.0030 ± .0002

.0071 ± .0006

.0096 ± .0010

.0035 ± .0003

.0051 ± .0004

RESCAL based model + cosine similarity;embedding_dim=50;entity_initializer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42;triples=all

.0049 ± .0004

.0116 ± .0009

.0149 ± .0006

.0061 ± .0005

.0086 ± .0006

DistMult based model + cosine similarity;embedding_dim=50;entity_initializer=None;entity_constrainer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42;triples=all

.0044 ± .0003

.0112 ± .0006

.0141 ± .0005

.0060 ± .0003

.0084 ± .0003

ComplEx based model + cosine similarity;embedding_dim=100;entity_initializer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42

.0048 ± .0005

.0114 ± .0009

.0141 ± .0012

.0064 ± .0005

.0088 ± .0007

RotatE based model + cosine similarity;embedding_dim=200;entity_initializer=None;relation_initializer=None;relation_constrainer=None;regularizer=None;epochs=25;seed=42;triples=all

.0042 ± .0005

.0103 ± .0014

.0133 ± .0013

.0064 ± .0010

.0087 ± .0012

The complete configuration for the second experiment with half the recommended embedding dimension size can be found in config_files\kge_parameters\kge_embedding-2.yml, and the full results besides averages and standard deviations can be found in experiment_results\kge_parameters\ml-100k_kge_embedding-2.csv:

Model

MAP@10

nDCG@10

Precision@10

Recall@10

F-score@10

TransE based model + cosine similarity;embedding_dim=75;scoring_fct_norm=1;entity_initializer=None;relation_initializer=None;relation_constrainer=None;regularizer=None;epochs=25;seed=42;triples=all

.0032 ± .0005

.0076 ± .0009

.0094 ± .0010

.0046 ± .0006

.0062 ± .0006

TransH based model + cosine similarity;embedding_dim=75;scoring_fct_norm=2;entity_initializer=None;relation_initializer=None;relation_regularizer=None;epochs=25;seed=42;triples=all

.0039 ± .0003

.0093 ± .0009

.0117 ± .0012

.0056 ± .0009

.0076 ± .0010

TransR based model + cosine similarity;embedding_dim=75;relation_dim=90;scoring_fct_norm=2;entity_initializer=None;entity_constrainer=None;relation_initializer=None;relation_constrainer=None;epochs=25;seed=42;triples=all

.0011 ± .0002

.0029 ± .0006

.0039 ± .0006

.0021 ± .0005

.0027 ± .0005

TransD based model + cosine similarity;embedding_dim=75;relation_dim=None;entity_initializer=None;entity_constrainer=None;relation_initializer=None;relation_constrainer=None;epochs=25;seed=42;triples=all

.0047 ± .0002

.0111 ± .0006

.0136 ± .0002

.0067 ± .0004

.0090 ± .0004

TuckER based model + cosine similarity;embedding_dim=100;relation_dim=None;dropout_0=0.3;dropout_1=0.4;dropout_2=0.5;apply_batch_normalization=True;relation_initializer=None;core_tensor_initializer=None;epochs=25;seed=42;triples=all

.0022 ± .0004

.0055 ± .0006

.0076 ± .0009

.0029 ± .0003

.0042 ± .0004

RESCAL based model + cosine similarity;embedding_dim=25;entity_initializer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42;triples=all

.0049 ± .0006

.0118 ± .0013

.0148 ± .0014

.0068 ± .0012

.0093 ± .0014

DistMult based model + cosine similarity;embedding_dim=25;entity_initializer=None;entity_constrainer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42;triples=all

.0042 ± .0005

.0104 ± .0007

.0131 ± .0008

.0059 ± .0009

.0081 ± .0009

ComplEx based model + cosine similarity;embedding_dim=50;entity_initializer=None;relation_initializer=None;regularizer=None;epochs=25;seed=42

.0054 ± .0003

.0126 ± .0011

.0149 ± .0015

.0069 ± .0007

.0095 ± .0009

RotatE based model + cosine similarity;embedding_dim=100;entity_initializer=None;relation_initializer=None;relation_constrainer=None;regularizer=None;epochs=25;seed=42;triples=all

.0043 ± .0004

.0104 ± .0008

.0131 ± .0011

.0060 ± .0010

.0082 ± .0011

ml-100k gnn_embedding

Ablation experiment changin only the embedding_dim parameter for the KGE component of the CKE and KGAT GNN models on the ml-100k dataset. The complete configuration for the first experiment with the recommended embedding dimension size can be found in config_files\kge_parameters\gnn_embedding-1.yml, and the full results besides averages and standard deviations can be found in experiment_results\kge_parameters\ml-100k_gnn_embedding-1.csv:

Model

MAP@10

nDCG@10

Precision@10

Recall@10

F-score@10

CKE;epoch=1000;kge_size=64;embed_size=150;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0169 ± .0007

.0382 ± .0015

.0421 ± .0019

.0274 ± .0013

.0332 ± .0014

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi;embed_size=150

.0179 ± .0012

.0407 ± .0022

.0438 ± .0022

.0284 ± .0014

.0344 ± .0015

The complete configuration for the second experiment with half the recommended KGE embedding dimension size for CKE and KGAT can be found in config_files\kge_parameters\gnn_embedding-2.yml, and the full results besides averages and standard deviations can be found in experiment_results\kge_parameters\ml-100k_gnn_embedding-2.csv:

Model

MAP@10

nDCG@10

Precision@10

Recall@10

F-score@10

CKE;epoch=1000;kge_size=64;embed_size=75;regs=[1e-05, 1e-05, 0.01];lr=0.0001

.0171 ± .0015

.0392 ± .0024

.0440 ± .0023

.0291 ± .0016

.0350 ± .0018

KGAT;n_layers=3;adj_type=si;adj_uni_type=sum;alg_typebi;embed_size=75

.0179 ± .0009

.0411 ± .0016

.0459 ± .0020

.0310 ± .0009

.0370 ± .0012