StreamRec: - VLDB

StreamRec: - VLDB

RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures Justin Levandoski Michael D. Ekstrand Michael J. Ludwig Ahmed Eldawy Mohamed F. Mokbel John T. Riedl Outline Overview of recommender systems

RecBench motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion

Recommender Systems Basic Idea Users: provide opinions on items consumed/watched/listened to The system: provides the user personalized suggestions for new items Collaborative Filtering (CF) ? ? ? Item-Based CF Model Building Similarity measures m1 m2

m3 similarity(m1,m3) similarity(m1,m2) Cosine distance Pearson correlation Spearman correlation Adjusted cosine distance ... .6 .8 .5 , , .8

.9 .5 , , .6 .7 .5 , , ... .3 , , .4

.4 , Item-Based CF Recommendations .6 .8 .5 , , .8 .9 .5 , ,

.6 .7 .5 , , .6 .9 .5 , , .6 .7 , .5

, .3 , , .4 .4 , .5 , .3 , Item-Based CF Recommendation ( ( ( .8

X 2 + .6 .9 X 2 + .5 X 4

+ .5 .7 ) / (.8 + .6) = 2.86 X 4) / (.9 + .5) = 2.71 X 2 ) / (.7 + .5) = 3.17 X 4 Outline

Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion RecBench Motivation Traditionally, recommender systems = customized software Example: Movielens Can we use relational DBMS primitives to build a recommender? No work has explored recommender system performance Performance has always been synonymous with quality [Our] solution is based huge amount of models and We have

chosen notontoadiscuss computation predictors which not be practical as part of aSuch commercial performance ofwould recommender algorithms. recommender system. However, this result is a direct performance is certainly important, and in the future we consequence of the nature and goal of the competition: obtain expect there to beaccuracy work on quality of time-limited the highest

possible at the any cost, disregarding completely and memory-limited recommendations. the complexity of the solution and the execution performance." BelKors Pragmatic ChaosFiltering Herlocker etTeam al. Evaluating Collaborative Winner of the ACM 2009TOIS Netflix Prize Recommender

Systems, 2004 RecBench Motivation Goals of RecBench: Prompt research community to explore space of alternative architectures for building recommender systems and study their performance Propose a benchmark to test performance (efficiency) of different system architectures Outline Overview of recommender systems

RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion RecBench Tasks http://www.movielens.org Task 1: Initialization Initialization Task Prepare system to start serving user recommendations Stored Recommendation Model Task 2: Pure Recommend

Pure Recommend Task Produce top-k recommendations from systems entire item pool Task 3: Filtered Recommend Filtered Recommend Task Produce top-k recommendations that match constraints on item metadata Task 4: Blended Recommend Blended Recommend Task Produce top-k recommendations based on blended text-search and recommendation score Task 5: Item Prediction Item Prediction Generate a users predicted rating for a target item

Task 6: Item Update Item Update Incorporate new item(s) into the system for recommendation Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion

Recommender System Architectures Unmodified DBMS MultiLens Hand-built system UMN recommender software suite (basis for MovieLens) Code optimized for item-based CF Uses DBMS for metadata and textsearch queries

PostgreSQL database Ratings relation: RecStore DBMS Database modified to optimize for fast ratings(usr,itm,rating) recommender model updates Model relation: SQL same as model(itm, rel_itm, sim) unmodified DBMS All tasks implemented in approach standard SQL (details in

Maintain partial paper) statistics to update rec. Metadata and text-search model incrementally as functionality built in new ratings arrive (e.g., selection) Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures

RecBench experimental results Conclusion Benchmark Setup Datasets MovieLens 10M movie ratings ~10K movies, ~70K users Architectures Netflix Challenge 100M movie ratings ~18K movies, ~480K users MultiLens Unmodified DBMS

RecStore DBMS Metric Metric is response time For recommendation queries, time is average of sample of 2.5% of total users broken down by quartile Same timing/sampling technique used for items for prediction task Tasks

Item-based collaborative filtering used as recommendation technique Tasks implemented specific to data from movie domain Run on 4-way 3 GHz Intel Xeon system with 48 GB or RAM Task 1: Initialization Task 1: build collaborative filtering model using movie ratings Netflix 100M MovieLens 10M 35000 60000 25000 50000 20000 40000 15000

10000 5000 3255 1709 0 Architecture 426 Time (sec) Time (sec) 30000 70000 30255

30000 61320 44666 23459 20000 10000 3371 0 Architecture Task 2: Pure Recommend Task 2: retrieve k recommendations using all possible movies Netflix 100M

MovieLens 10M DBMS/RecStore MA MultiLens RecStore MI DBMS/RecStore MA MultiLens 14 10 Time (ms) Time (ms) 12 8 6 4 2 0

quartile 1 quartile 2 quartile 3 quartile 4 User Quartile 50 45 40 35 30 25 20 15 10 5 0 RecStore MI quartile 1 quartile 2 quartile 3 quartile 4 User Quartile

Task 3: Filtered Recommend Task 3: retrieve k recommendations of only 1990s comedies Netflix 100M MovieLens 10M 20 18 16 14 12 10 8 6 4 2 0 RecStore MI

DBMS/RecStore MA MultiLens RecStore MI 16 14 Time (ms) Time (ms) DBMS/RecStore MA MultiLens 12 10 8 6 4 2

quartile 1 quartile 2 quartile 3 quartile 4 User Quartile 0 quartile 1 quartile 2 quartile 3 quartile 4 User Quartile Outline Overview of recommender systems RecBench benchmark tasks Recommender system architectures RecBench experimental results

Conclusion Future Work Just scratched the surface on recommender system architecture and performance work More performance studies Multi-user experiments Different recommendation techniques Recommender system architectures Hybrid systems Recommendation-specific operators [Koutrika et al, SIGMOD09] Conclusion and Summary http://lenskit.grouplens.org Recbench available as part of LensKit project from University of Minnesota Questions?

Recently Viewed Presentations

  • Chapter 23: Antitrust Law and Promoting Competition  2013

    Chapter 23: Antitrust Law and Promoting Competition 2013

    The Sherman Antitrust Act. Major Provisions of the Sherman Act. Every contract, combination in the form of trust or otherwise, or conspiracy, in restraint of trade or commerce among the several States, or with foreign nations, is hereby declared to...
  • ECO 365 - Intermediate Microeconomics

    ECO 365 - Intermediate Microeconomics

    Fixed inputs - fixed even if y=0. Quasi-fixed inputs (if y=0 then x=0 but if y>0 then x is fixed). Short-run π maximization Assume only 1 output Y and two inputs L & K K is fixed => in short-run...
  • Lucent Technologies Bell Labs Innovations Lucent Bell Labs

    Lucent Technologies Bell Labs Innovations Lucent Bell Labs

    Lucent Technologies was formed in Oct. 1996 when AT&T separated into three completely independent companies. Systems and equipment business along with 85% of the Bell Labs moved into Lucent. Lucent Technologies (China) is the holding company handling all Lucent's business...
  • Outer Westbrook Street Rezoning Planning Staff Presentation Revised

    Outer Westbrook Street Rezoning Planning Staff Presentation Revised

    Currently the FHO North of Westbrook St would allow 53 manufactured house lots. Or 16 single family under R-1. Currently a mix of land in private ownership, City ownership, Turnpike and Utility abutting the Fore River Sanctuary and Tech Park
  • The Color Wheel

    The Color Wheel

    Where does color come from? A ray of light is the source of all color. Without light, color does not exist. Light is broken down into colors of the spectrum. You can often see a variety of colors when you...
  • FileNewTemplate

    FileNewTemplate

    SB 0428 - For three years beginning July 1, 2017, those who have retired as active classroom teachers and who have not been employed by any public school for at least one year postretirement will be eligible to be re-employed...
  • Honors Pre-calculus

    Honors Pre-calculus

    Special Journals General Journal Subsidiary Ledgers General Ledger Accounting II Essential Learning Expectations-Chapters 1 and 2: Special journals are multicolumn journals used to record specific types of business transactions. A subsidiary ledger contains detailed data summarized in a controlling account...
  • VERTEBRATES

    VERTEBRATES

    Animals with backbones. Animals with backbones or vertebrates are classified into 5 subgroups. Fish is the largest of vertebrates. FISH. Breathe through a set of gills. Covered with hard scales. Reproduce by laying soft, non-shelled eggs. Cold blooded.