Panagiotis Ipeirotis

Panagiotis G. Ipeirotis (also known as Panos Ipeirotis; born May 3, 1976, in Serres, Greece) is a Greek-American computer scientist and the Merchants' Council Professor of Technology and Business at the New York University Stern School of Business. His research focuses on data mining, crowdsourcing, human computation, and the economics of online information systems.

Ipeirotis received the Lagrange Prize in Complex Systems (2015)

In addition to his academic career, Ipeirotis co-founded the AI consulting firm Detectica in 2015, which was acquired by Compass, Inc. in 2019. He has also held research positions at Google (2013–2014) and at Meta's Reality Labs division (2024–2025). Later that year, he represented Greece at the 26th International Chemistry Olympiad in Oslo, Norway.

Education Ipeirotis earned his Diploma in Computer Engineering and Informatics (CEID) from the University of Patras in 1999. and is an associated faculty member at the NYU Center for Data Science.

Ipeirotis has served as Program Co-Chair for the 13th ACM Conference on Electronic Commerce in 2012, as Technical Program Co-Chair for The Web Conference 2018 (WWW 2018), and as an Area Chair for the AAAI Conference on Artificial Intelligence in 2021 and 2023. He co-organized the Human Computation Workshop series from its founding edition in 2009 (co-located with SIGKDD) through 2012, and served as General Co-Chair of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2015, after the series had grown into a stand-alone AAAI conference. He has served on the editorial boards of Management Science, IEEE Transactions on Knowledge and Data Engineering, the INFORMS Journal on Computing, and the ACM Journal of Data and Information Quality, and was a founding co-Editor-in-Chief of the journal Collective Intelligence, launched in 2022 as a collaboration between SAGE Publications, the Association for Computing Machinery, and Nesta.

Ipeirotis has been the subject of media profiles in business and Greek-language outlets. Bloomberg Businessweek profiled him in 2011, describing his work on technology that combines human and machine intelligence, and again in 2013, in a feature that called him the "data dude" of business analytics. A 2014 Kathimerini interview discussed his career and conditions in Greek higher education.

Academic lineage and doctoral students Ipeirotis's doctoral advisor Luis Gravano was himself a student of Hector Garcia-Molina at Stanford. According to the Mathematics Genealogy Project, Ipeirotis has advised seven doctoral students at NYU and Columbia.

His doctoral students include Beibei Li (now at Carnegie Mellon University's Heinz College) and Marios Kokkodis, both of whom subsequently received INFORMS Information Systems Society early-career awards — the Sandra A. Slaughter Early Career Award in 2019 and the Gordon B. Davis Young Scholar Award in 2020. Former postdoctoral researcher Djellel Difallah is Assistant Professor at New York University Abu Dhabi.

Research Ipeirotis's research spans text mining, crowdsourcing, and the economics of online information systems. His work has been published in venues such as Management Science, Information Systems Research, and IEEE Transactions on Knowledge and Data Engineering. which the thesis later generalized into the QProber system using machine learning classifiers to derive the probing queries. In parallel, Ipeirotis contributed to research on data integration: a 2001 paper at the International Conference on Very Large Data Bases (VLDB), co-authored with Gravano, H. V. Jagadish, Nick Koudas, S. Muthukrishnan, and Divesh Srivastava, showed how approximate string joins — matching slightly different versions of the same entity across databases — could be implemented entirely within standard relational database systems, without specialized indexes.

Building on this foundation, Ipeirotis published two award-winning papers on modeling and querying text databases. A 2005 study applied survival analysis techniques to model how the content of 152 web databases changed over a 52-week period, winning the Best Paper Award at the 21st IEEE International Conference on Data Engineering. co-authored with Gabriele Paolacci and Jesse Chandler, provided methodological guidance for using the platform — commonly abbreviated MTurk — in behavioral research.

A 2010 study by Ipeirotis estimated that approximately 40% of tasks posted by new requesters on MTurk involved spam or low-quality content, drawing media coverage.

His 2010 demographic analysis of the MTurk workforce provided early data on worker motivations, geographic distribution, and income levels, and has been cited by researchers studying crowdsourcing platforms. A follow-up study with former postdoctoral researcher Djellel Difallah and Elena Filatova, presented at the ACM International Conference on Web Search and Data Mining in 2018, tracked changes in the MTurk workforce over an eight-year period. Media outlets have continued to draw on this body of work as concerns about the human labor underlying machine learning training data have grown, including coverage in Wired and The New York Times on automated responses on MTurk and a 2026 Marketplace interview in which Ipeirotis discussed CAPTCHA completion and other routine online interactions as unrecognized AI-training-data labor. The Financial Times cited his demographic and wage data on Mechanical Turk workers in a 2016 examination of the human workforce behind commercial artificial intelligence, and The Wall Street Journal quoted him in 2017 on the continued reliance of large companies on human workers within their AI systems. In 2020, Wired quoted him on an apparent influx of new workers onto Mechanical Turk during the COVID-19 pandemic, including workers based in Canada, Italy, and Brazil.

This line of work also led to the development of quality management techniques for crowdsourcing. The 2008 paper "Get Another Label?," co-authored with Foster Provost and Victor S. Sheng, addressed the problem that training machine learning classifiers on labels from non-expert workers degrades model accuracy when those labels are noisy. The paper studied whether repeated labeling of uncertain items could improve classifier performance under noisy labels, and proposed selective repeated-labeling strategies for crowdsourced annotation. It received the Best Paper Runner-Up Award at the 2008 ACM SIGKDD Conference on Knowledge Discovery and Data Mining and the SIGKDD Test of Time Award in 2020.

ProPublica drew on Ipeirotis's research and blog posts in publishing its "Guide to Mechanical Turk" for investigative reporters, citing his work on data quality through multiple worker verification. Ipeirotis has also commented on the use of crowdsourcing for content moderation; in a 2012 Foreign Policy article on image moderation, he described comparing the judgments of moderators from different regions as a way to reduce cultural bias.

To support this research, Ipeirotis developed "MTurk Tracker," a set of tracking tools for the MTurk platform. The first tool tracked workforce demographics using capture-recapture techniques to estimate the size and composition of the workforce in near real-time. The second tool periodically downloaded all tasks posted on the platform, enabling analysis of the crowdsourcing marketplace's size and activity. The Pew Research Center's 2016 report "Research in the Crowdsourcing Age" drew on data from the task tracker, explicitly crediting "the online tool mturk-tracker, which is run by Dr. Panagiotis G. Ipeirotis." The 2019 book Ghost Work, by Mary L. Gray and Siddharth Suri, drew on Ipeirotis's worker-demographics research to estimate the size of the platform's active workforce.

During a 2013–2014 sabbatical at Google, Ipeirotis worked on crowdsourcing methods for improving the Google Knowledge Graph. He co-developed "Quizz," a gamified system that presented data collection as trivia quizzes targeted by domain expertise, using calibration questions with known answers to assess user reliability in real-time. In a related paper with Chun How Tan, Eugene Agichtein, and Evgeniy Gabrilovich, Ipeirotis examined methods for predicting contribution quality in knowledge-base construction and curation.

Ipeirotis also applied crowdsourcing methods to building training data for computer vision systems. With Pietro Perona and Serge Belongie — co-recipients with Ipeirotis of a 2012 Google Focused Research Award — he co-authored a 2015 paper at the IEEE Conference on Computer Vision and Pattern Recognition describing the construction of a large-scale dataset for fine-grained bird species recognition using citizen-scientist labels.

In a 2015 article in the ACM Journal on Data and Information Quality, co-authored with Josh Attenberg and Foster Provost, Ipeirotis introduced "Beat the Machine," a framework in which crowdworkers were rewarded for finding examples that a trained classifier predicted incorrectly. The approach was proposed as a method for surfacing the "unknown unknowns" — systematic blind spots — of machine learning models.

Data quality and record linkage Ipeirotis has published research on duplicate record detection and data quality. The 2007 survey "Duplicate Record Detection: A Survey," co-authored with Ahmed Elmagarmid and Vassilios Verykios, appeared in IEEE Transactions on Knowledge and Data Engineering and reviewed record linkage techniques across the database, statistics, and machine learning literatures.

Online reputation and user-generated content As part of the EconoMining project, Ipeirotis and Anindya Ghose quantified the pricing power derived from textual content in product reviews; the work was discussed in the Financial Times. A related 2011 study found that the quality of spelling and grammar in product reviews affects perceived helpfulness and product sales. The finding was discussed in Forbes, Harvard Business Review, Slate, and the Freakonomics blog.

Product and travel search With Anindya Ghose and doctoral student Beibei Li, Ipeirotis developed text-mining and econometric methods for ranking products and hotels on online platforms. A 2011 paper introduced a theoretical model for product search that combined consumer review text with product characteristics to rank items by their predicted economic utility, receiving the Best Paper Award at the 20th International World Wide Web Conference. A 2012 study in Marketing Science extended the framework to hotel ranking on travel search engines, combining mining of online reviews with crowdsourced ratings of hotel attributes. A 2014 study in Management Science analyzed how the ordering of search results affects consumer click-through and platform revenue.

Online labor markets With doctoral student Marios Kokkodis, Ipeirotis studied reputation systems and hiring dynamics in online labor markets such as oDesk (now Upwork). A 2015 study at the ACM International Conference on Web Search and Data Mining modeled employer hiring decisions using features such as worker reputation, prior experience, and bid characteristics. Their work on reputation transferability modeled how a freelancer's performance ratings in one task category can predict outcomes in another, analyzing over one million transactions; it was published in Management Science in 2016. A subsequent study applied reinforcement learning to recommend career paths to freelancers based on dynamic market demand, finding that the framework could increase both marketplace revenue and worker wages. They also developed recommender systems for matching job applicants to employers on freelancing platforms.

Applied research and industry

Early industry work (2009–2014) Beginning in 2009, Ipeirotis served as an early advisor to Integral Ad Science (originally AdSafe Media), contributing to the development of machine learning systems for detecting inappropriate web content and advertising fraud. In 2011, working with AdSafe engineers, he identified a click fraud scheme that used hidden iframes to generate fraudulent ad impressions, which he termed "traffic laundering." The investigation, which MIT Technology Review headlined as "A Web Scam That Makes $500,000 a Month," was also reported by The Wall Street Journal and led to an FBI referral.

He also served as Academic-in-Residence at Upwork (then oDesk) in 2012 and as a Visiting Scientist at Google from 2013 to 2014. Drawing on his empirical research from Mechanical Turk and oDesk, the report estimated the online outsourcing market would grow to $15–25 billion by 2020 and identified barriers preventing workers in the Global South from accessing these jobs, including lack of reliable internet infrastructure and the "reputation cold start" problem. The report has been cited in subsequent policy work by the RAND Corporation on online outsourcing as a youth-employment pathway in Indonesia.

Detectica and Compass (2015–2022) In 2015, Ipeirotis co-founded Detectica with Foster Provost and Josh Attenberg, offering AI strategy consulting and machine learning solutions for business applications. The company developed AI-driven compliance monitoring systems for financial institutions.

Meta Reality Labs (2024–2025) From 2024 to 2025, Ipeirotis was a visiting researcher at Meta's Reality Labs division, working on machine learning deployment for wearable devices.

Public engagement and pedagogy Since 2008, Ipeirotis has maintained the blog "A Computer Scientist in a Business School," which has served as the original venue for several posts that subsequently drew independent media coverage, including his 2011 commentary on academic integrity and his 2025 description of an AI-driven oral examination format.

Academic integrity debates In July 2011, Ipeirotis published a post on his blog, titled "Why I will never pursue cheating again," describing his experience using Turnitin software to detect that about 20 percent of the students in his roughly 100-person "Information Technology in Business and Society" course had plagiarized. The post criticized the administrative burden placed on faculty who enforce academic integrity policies and drew coverage in higher-education media, including Inside Higher Ed and The Chronicle of Higher Education. Ipeirotis temporarily removed the post following concerns raised about whether quoted student communications could implicate the Family Educational Rights and Privacy Act (FERPA). NYU Stern vice dean Ingo Walter responded that faculty are not penalized for enforcing academic integrity and that student evaluations from such cases are excluded from review.

Security vulnerability discovery In April 2012, Ipeirotis discovered a vulnerability in Google Spreadsheets that he termed a "Denial of Money" attack. While running a crowdsourcing experiment that required image classification, he uploaded 25,000 image URLs (hosted on his Amazon S3 bucket) into a Google Spreadsheet. Google's Feedfetcher bot repeatedly re-downloaded all images from his S3 bucket every time the spreadsheet was processed, generating a high volume of outgoing bandwidth. Ipeirotis reported receiving an Amazon Web Services bill of $1,177.76—approximately ten times his normal monthly rate.

Coverage of the incident described it as illustrating a potential attack vector: a malicious actor could create a Google Sheet with thousands of links to a victim's large files, causing Google's servers to repeatedly request the files and resulting in either financial damage or a denial of service. The discovery was covered in technology publications including Wired and The Economist. Amazon subsequently forgave the charges, and Google investigated the behavior. An Associated Press feature in April 2026 reported on the approach as part of broader coverage of U.S. colleges adopting oral examinations in response to generative AI, quoting Ipeirotis on his rationale: "I don't trust written assignments anymore to be the result of actual thinking." Ipeirotis and Konstantinos Rizakos described the system's design and evaluation in a 2026 preprint, in which the examinations used a multi-agent workflow and a panel of large language models to grade the resulting transcripts. In a student survey reported by the NYU student newspaper Washington Square News, 70 percent of respondents agreed that the oral format better assessed their comprehension of course material, while 83 percent found it more stressful than written exams. The approach was also discussed on the academic-philosophy site Daily Nous and the education-technology podcast EdTechnical.

Awards and honors

2020: ACM SIGKDD Test of Time Award for Research (with Foster Provost and Victor S. Sheng). The Test of Time Award honors papers from approximately ten years prior that have had lasting impact on the field of knowledge discovery and data mining. The award recognized their 2008 paper "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers" "for their approach to selective acquisition of multiple labels." 2015: Lagrange Prize in Complex Systems (shared with Jure Leskovec), awarded by the CRT Foundation and ISI Foundation. The prize citation recognized Ipeirotis's "pioneering work in the field of crowdsourcing and human computation" and research "[combining] economic, social psychology and automatic text analysis methods to quantify the economic value of the content generated by users on the Internet." 2014: Best Paper Award, Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP), for "STEP: A Scalable Testing and Evaluation Platform". 2014: INFORMS Information Systems Society (ISS) Best Paper Award (Management Science), for "Deriving the Pricing Power of Product Features by Mining Consumer Reviews" (2011). 2012: Google Focused Research Award ($1.5 million), with Serge Belongie and Pietro Perona. 2011: Best Paper Award, 20th International World Wide Web Conference (WWW 2011), for "Towards a Theory Model for Product Search". 2008: Best Paper Runner-Up Award, 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2008), for "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers" (with Foster Provost and Victor S. Sheng). 2006: Best Paper Award, ACM International Conference on Management of Data (SIGMOD 2006) — "To Search or to Crawl? Towards a Query Optimizer for Text-Centric Tasks". 2005: Best Paper Award, 21st IEEE International Conference on Data Engineering (ICDE 2005), for "Modeling and Managing Content Changes in Text Databases".

Selected publications

(KDD Test of Time Award, 2020)