Asking AI to help me decide how to vote for the SF mayor

Can AI help you make better decisions?

I asked Gemini to help me decide who to vote for in the upcoming SF mayoral election by reading through some relevant press coverage. I asked it to focus on four issues and make a recommendation on each issue: housing, homelessness, public safety, and downtown revival.

Here's what I have so far:

Housing

London Breed: Prioritizes "Housing for All," streamlining approvals, rezoning, and supporting housing bonds. (Source)

Mark Farrell: Limited information available. General focus on city revitalization and budget, which could impact housing. (Source)

Asha Safai: Limited policy details available. More research needed. (Source)

Dan Lurie: Focuses on building affordable housing, citing Tipping Point Community experience. (Source)

Aaron Peskin: Advocates for rent control expansion, shelter beds, and neighborhood development. (Source)

Recommendation: Breed/Lurie. Breed's experience and plans are strong. Lurie's affordable housing focus is promising but needs more detail.

Homelessness

London Breed: Increased shelter capacity, helped people exit homelessness, focused on encampments and conservatorship laws. (Source)

Mark Farrell: Wants to clear encampments, increase police hiring, address the opioid crisis. (Source)

Asha Safai: Limited detailed proposals available. (Source)

Dan Lurie: Plans to build more shelter beds, tiny homes, expand Homeward Bound, and designate RV parking areas. (Source)

Aaron Peskin: Supports more shelter and affordable housing, criticizes sweeps. (Source)

Recommendation: Lurie/Peskin. Lurie's plan is detailed. Peskin's focus on long-term solutions like affordable housing is commendable.

Public Safety

London Breed: Focused on reducing crime and drug overdoses. (Source)

Mark Farrell: Wants to fire police chief, hire more officers, deploy National Guard for drug dealing. (Source)

Asha Safai: Limited detailed proposals available. (Source)

Dan Lurie: Limited detailed proposals available. (Source)

Aaron Peskin: Limited detailed proposals, but focuses on supporting neighborhoods and community. (Source)

Recommendation: Farrell/Breed. Farrell's approach is aggressive but raises militarization concerns. Breed's experience may be preferred.

Downtown Revival

London Breed: "30 by 30" plan to attract residents and students, fill storefronts, support businesses. (Source)

Mark Farrell: Limited information, but stresses economic recovery. (Source)

Asha Safai: Limited detailed proposals available. (Source)

Dan Lurie: Limited detailed proposals available. (Source)

Aaron Peskin: Limited information, general focus on small businesses and neighborhoods. (Source)

Recommendation: Breed. "30 by 30" offers a concrete vision, though its feasibility needs review.

Endorsements

London Breed: SF Democratic Party, various officials, community leaders. (Source)

Mark Farrell: GrowSF, SF Firefighters, TogetherSF Action, Neighbors for a Better San Francisco. (Source)

Asha Safai: SEIU Local 87, UFCW Locals 5 and 648. (Source)

Dan Lurie: GrowSF, various officials, community leaders. (Source)

Aaron Peskin: SEIU 1021, various unions, progressive organizations. (Source)

If you're curious, the prompt is something along the lines of:

Please construct a table helping me compare the major candidates running for mayor of San Francisco on major issues. Consider the following candidates: London Breed, Mark Farell, Asha Safai, Dan Lurie, and Aaron Peskin. Focus on housing, homelessness, public safety, and downtown revival as the 4 major issues. Put each candidate in a separate column and each issue in a separate row of the table. For each issue, produce a final column titled “Recommendation” stating who might be the best candidate on that issue (and why). Finally, include a row called "Endorsements" and list any organizations or major public figures that have endorsed the candidate. Please cite your sources and list the URLs you are citing after the table.

How to get paid less than minimum wage

Netflix has an ad-supported Basic tier for $6.99/mo. The version without ads costs $9.99/mo. The average US adult subscriber spends 30 minutes a day watching Netflix. That's about 15 hours. Netflix claims to show about 4 minutes of ads per hour of viewing. That's 4 * 15 = 60 minutes or 1 hour a month. So you can pay an average US adult $3.00/hr to watch ads for about an hour a month. 

The ad-supported version is a good deal only if you spend very little time on Netflix. In fact, if you value your ad-view time at:

• the federal minimum wage ($7.25), you should watch fewer than 6.2 hours a month on Netflix Basic with Ads.
• the California minimum wage ($15.00), you should watch fewer than 3 hours a month (6 minutes per day) on Netflix Basic with Ads.

OK, I'm now going back to watching TikTok videos for $0.00/hr.

Building a great home espresso setup for $600

If you're an espresso drinker, and have wondered if it is possible to make great espresso at home without spending too much money or counter-space, read on.

Why bother?

I live in San Francisco within a 20-minute walk of ~10 coffee shops that make great espresso. For years that was a good enough reason to just make pour-overs at home and not get a home espresso setup. I occasionally wondered how a coffee might taste as espresso instead of pour-over. If you’ve had the same thought, then it might be worth getting a home espresso setup so you have the freedom to make espresso with whatever coffee you buy. I tend to like lighter roasted natural processed coffee that isn’t a very popular choice for espresso at most cafes.

The general advice is to avoid making espresso at home. It can be a nightmare. However, if you don’t need to make milk drinks, are willing to put in a little elbow grease, and don’t want to commit a lot of counter space or money, here’s the setup I recommend. This has served me well through the pandemic, and has yielded many delicious espressos at home for a range of different beans.

Hand Grinder

1ZPresso JE Plus Manual Hand-Grinder. ($239) A hand-grinder is cheaper, and all the money is going into getting high quality burrs. It takes a little over a minute to grind 15g of beans .The JMax  is a little bit cheaper at $199 and is also excellent for espresso if you prefer a unimodal style vs. creamier shots. See reviews for similar options (1, 2, 3). There are several good options at the $250 price-point. 

Espresso Machine

Cafelat Robot ($310). With a manual machine, you’re not dealing with parts that break and/or need maintenance. It also does not need to be plugged into an outlet. The machine is beautiful, and made with fairly high-quality parts. The lever needs to be pushed with some force, but it is something most adults can manage fairly easily. See James Hoffman’s review here.

Scales

You need fairly accurate scales to measure your coffee and the yield (so you know when to stop extracting). These are $12 on Amazon and work just fine.

 

That adds up to about $560 + taxes. In the US, that’ll likely run you a bit more than $600. That’s it. You need a kettle that can boil water, and you should be able to make very good espresso at home. Enjoy!

KDD 2018: Anatomy of a Privacy-Safe Large-Scale Information Extraction System Over Email

I'm excited to share my team's work that will be presented at KDD 2018. Information extraction can be a hard problem, and doing that in a privacy-safe manner from email harder yet. Our paper lays out how to build such a system.

Abstract from the paper:

Extracting structured data from emails can enable several assistive experiences, such as reminding the user when a bill payment is due, answering queries about the departure time of a booked flight, or proactively surfacing an emailed discount coupon while the user is at that store. This paper presents Juicer, a system for extracting information from email that is serving over a billion Gmail users daily. We describe how the design of the system was informed by three key principles: scaling to a planet-wide email service, isolating the complexity to provide a simple experience for the developer, and safeguarding the privacy of users (our team and the developers we support are not allowed to view any single email). We describe the design tradeoffs made in building this system, the challenges faced and the approaches used to tackle them. We present case studies of three extraction tasks implemented on this platform—bill reminders, commercial offers, and hotel reservations—to illustrate the effectiveness of the platform despite challenges unique to each task. Finally, we outline several areas of ongoing research in large-scale machine-learned information extraction from email.

ICDE 2018: Recommendations for All : Solving Thousands of Recommendation Problems Daily

I'm excited to share that our paper on scaling a recommendation servicewas accepted to ICDE 2018! An ML+Systems perspective.
Recommender systems are a key technology for many online services includinge-commerce, movies, music, and news. Online retailers use product recommender systemsto help users discover items that they may like. However, building a large-scaleproduct recommender system is a challenging task. The problems of sparsity andcold-start are much more pronounced in this domain. Large online retailers have usedgood recommendations to drive user engagement and improve revenue, but the complexityinvolved is a roadblock to widespread adoption by smaller retailers.
In this paper, we tackle the problem of generating product recommendations for
tens of thousands of online retailers. Sigmund is an industrial-scale system forproviding recommendations as a service. Sigmund was deployed to production in early2014 and has been serving retailers every day. We describe the design choices that wemade in order to train accurate matrix factorization models at minimal cost.We also share the lessons we learned from this experience -- both from a machinelearning perspective and a systems perspective. We hope that these lessons areuseful for building future machine-learning services.

WWW 2018 : Hidden in Plain Sight – Classifying Emails Using Embedded Image Contents

I'm excited to share that my team's paper on classifying emails using embedded image contents has been accepted to WWW 2018 (aka The Web Conference 2018).

In this paper, we tackle the problem of extracting information from commercial emails promoting an offer to the user.  The sheer number of these promotional emails makes it difficult for users to read all these emails and decide which ones are actually interesting and actionable.  Extracting information  enables an email platform to build several new experiences that can unlock the value in these emails without the user having to navigate and read all of them. For instance, we can highlight offers that are expiring soon, or display a notification when there’s an unexpired offer from a merchant if your phone recognizes that you are at that merchant’s store.

A key challenge in extracting information from such commercial emails is that they are often image-rich and contain very little text. Training a machine learning (ML) model on a rendered image-rich email and applying it to each incoming email can be prohibitively expensive. In this paper, we describe a cost-effective approach for extracting signals from both the text and image content of commercial emails in the context of a free email platform that serves over a billion users around the world. The key insight is to leverage the template structure of emails, and use off-the-shelf OCR techniques to obtain the text from images to augment the existing text features offline. Compared to a text-only approach, we show that we are able to identify 9.12% more email templates corresponding to ~5% more emails being identified as offers. Interestingly, our analysis shows that this 5% improvement in coverage is across the board, irrespective of whether the emails were sent by large merchants or small local merchants, allowing us to deliver an improved experience for everyone.

KDD 2017 Teaser Video and Paper

Check out the teaser video for our paper at KDD 2017!  The PDF of the paper is available here at the Google Research web site.

Drive Quick Access at KDD 2017

Our paper, "Quick Access: Building a Smart Experience for Google Drive" was accepted for presentation at KDD 2017. The development of the Quick Access feature illustrates many general challenges and constraints associated with practical machine learning such as protecting user privacy, working with data services that are not designed with machine-learning in mind, and evolving product definitions. We believe that the lessons learned from this experience will be useful to practitioners tackling a wide range of applied machine-learning problems. We're looking forward to sharing this with the KDD community.

Quick Access on Drive: How Machine Learning can save you time

Check out my blog post on the Google Research blog about how ML saves you time with Quick Access on Drive.

Quick Access on Drive (Web and iOS)

Quick Access: Machine Learning in Google Drive

I'm very excited about a new feature in Drive that our team in Research/Machine Intelligence helped launch today: Quick Access for Drive.

Quick access uses Google Machine learning on your Drive Activity, and your daily interactions to bring you the documents you need before you try to search/navigate to it. Our user-research shows that it shaves 50 percent off the average time it takes to get to the right file. Quick Access is available for G Suite customers on Android. If you're interested in all the new intelligent features in Google Apps, check out this post from Prabhakar Raghavan, the VP for Google Apps.

UDF Centric Workflows

A recent PVLDB paper titled "An Architecture for Compiling UDF-Centric Workflows" describes some pretty exciting advances in the state of the art for data analysis. Using a system called Tupleware, the authors of the paper show that for datasets that fit into the memory of a small cluster, it is possible to extract far more performance from the hardware than is commonly provided on platforms like Spark. For instance, on a K-Means task, Tupleware was faster than Spark by 5X on a single node, and by nearly 100X on a 10-node cluster. For small datasets, that probably means a 1000x (3 orders of magnitude) improvement over Hadoop.

The dramatic performance improvement come from leveraging several known techniques from the literature on main-memory database systems and compilers. Tupleware relies on combining both high-level optimizations common in relational optimizers (projection and selection pushdown, join order optimization) with optimizations considered by recent main-memory systems like pipelining operators vs. vectorized simple operators. Several techniques and heuristics are described in the paper that produce this combined performance improvement.

Tupleware generates and compiles a program for each workflow, and leaning on the LLVM compiler framework lets it use many techniques like inlining UDFs, SIMD vectorization, generated implementations for context variables, etc. In the distributed setting, Tupleware communicates at the level of blocks of bytes while Spark uses fairly expensive Java serialization that consumes a fair fraction of the runtime for distributed execution. Tupleware does not describe any support for fault-tolerance, but with the huge performance improvement over Spark for many applications, simply restarting the job may be reasonable until we get to a certain job size.

Many researchers have pointed out that the fault-tolerance trade-offs that are right for analysis of large datasets (hundreds of terabytes) are not the same as the ones for smaller datasets (few terabytes). Glad to see data management research that's highlighting this well beyond the "X is faster than Hadoop for T".

Insurance and Self-Driving Cars

Metromile (a car insurance company that's pioneering a pay-as-you-go model) recently produced an analysis of what their monthly insurance rates would be for a self-driving car based on the accident stats that Google released. Check out the expected savings on a variety of models:

No surprise that insurance companies like self-driving cars -- for the kinds of applications we've seen these cars used for, they are much safer than the typical (distracted) American driver. This is one of those world-changing technologies I'm eagerly looking forward to!

Google Could Bigtable : NoSQLaaS

I'm very excited that Google announced Cloud Bigtable. You get a managed instance of Bigtable with an open-source API (there's an HBase client) at a great price! Bigtable's HBase APIs are fairly simple and easy to use, and we seem to have mapped nearly all of the features in the client to Bigtable.

Combined with Google Cloud Dataflow, which lets you write Flume jobs over your data for analytics, I think Google's Cloud Platform is very compelling for data intensive applications. I'm a huge fan of Flume, and am absolutely convinced it provides a huge boost to developer productivity compared to writing direct mapreduces. I think the platform finally has all the pieces needed for big data applications far quicker than rolling-your-own Hadoop+Mapreduce+HBase+Hive/Pig/Scalding stack in the cloud. I'm excited to see how developers will use this platform.

On the dangers of AUC

In most applied ML projects I've been involved in people rarely ever use AUC (the Area Under the ROC Curve) as a performance measure for a classifier. However, it is very commonly reported  in many academic papers. On the surface, the AUC has several nice properties: it is threshold free; so you can compute AUC without having to pick an classification threshold, it is insensitive to changes in the ratio of positive to negative examples in the evaluation set, and it has the nice property that a random classifier has an AUC of 0.5.  

It is easy to see that AUC can be misleading when used to compare two classifiers if their ROC curves cross. Classifier A may produce a higher AUC than B, while B performs better for a majority of the thresholds with which you may actually use the classifier. And in fact empirical studies have shown that it is indeed very common for ROC curves of common classifiers to cross. There are also deeper reasons why AUC is incoherent and therefore an inappropriate measure (see references below).

In this post however, I'll describe the common scenario of training a classifier on highly imbalanced problems (where the data is such that one class has many more examples than another). We only discuss binary classification in this post, so there are only two classes. The AUC measure, is of course, insensitive to class imbalance, but produces rather misleading results. Consider a problem where we have a large number of negative examples and a small number of positive examples. Further, a large number of negative examples are “easy” -- that is you can get most of them right without much effort. This is a fairly common situation with problems like spam/fraud detection in various domains. In such a situation, a classifier that gets the “easy” negatives right, but offers random predictions for the rest, actually does really well on AUC:

(Example below uses the AUC  library in R)

n <- 5000 # For this scenario, we use 80% easy negatives (value 0), and the remaining 20% are evenly balanced between positives and negatives (values 1, and -1). d <- sample( c(numeric(8), -1, 1), n, replace=TRUE) # Labels maps all negatives to 0, all positives to 1. labels <- factor((d > 0) * 1) # We predict all easy negatives correctly, and random predictions for the rest. rand_preds <- runif(n) * abs(d) # All easy negatives correct, better predictions for rest with some noise. good_preds <- d*.1 + rand_preds # All easy negatives correct. Better, less noisy predictions for the rest. better_preds <- d*.3 + rand_preds

Now examine the AUC-ROC for these predictors:

> auc(roc(rand_preds, labels)) [1] 0.9430441 > auc(roc(good_preds, labels)) [1] 0.9628872 > auc(roc(better_preds, labels)) [1] 0.9896737

A couple of observations:

1. While technically ‘rand_preds’ only gets the easy examples right, and is no better than random on the ones that matter, it produces a very high AUC of 0.94.
2. Because of the large number of true negatives, the differences between the random, good, and better predictors are very small (of the order of 2%-3% change in AUC).

If you increase the imbalance to be 98% easy negatives, and 2% hard positives and negatives, the AUC differences are even smaller:

# 98% easy negatives: d <- sample( c(numeric(98), -1, 1), n, replace=TRUE) > auc(roc(rand_preds, labels)) [1] 0.995519 > auc(roc(good_preds, labels)) [1] 0.9974802 > auc(roc(better_preds, labels)) [1] 0.9996038

If your problem-specific metric is something like Precision@Recall=80%, these classifiers behave *very* differently:

#Plot Precision-Recall curves (using the ROCR library) plot(performance(prediction(better_preds, labels), "prec", "rec"), col='green') plot(performance(prediction(good_preds, labels), "prec", "rec"), col='blue', add=TRUE) plot(performance(prediction(rand_preds, labels), "prec", "rec"), col='red', add=TRUE)

Precision-Recall Curves for the three classifiers

Precision@Recall=80% for the ‘random’ predictor is ~0.5 (as expected), for the ‘good’ predictor is ~0.65 and the ‘better’ predictor is ~0.9. These differences are huge compared to the AUC numbers which only differ in the third significant digit.

Of course, everybody would recommend using a problem-specific measure ahead of a more generic measure like AUC, the point of this post is to argue that even if you're only using AUC for a quick-and-dirty comparison you shouldn't look at a high value in imbalanced problems and be impressed with the performance.

Some interesting references:
Jesse Davis and Mark Goadrich. The relationship between Precision-Recall and ROC curves. ICML 2006

D.J. Hand, C. Anagnostopoulos, When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance? Pattern Recognition Letters, Volume 34, Issue 5, 1 April 2013, Pages 492–495

David J. Hand, Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning October 2009, Volume 77, Issue 1, pp 103-123

WSDM 2015

I recently got back from WSDM which ended up being a very interesting conference with some great talks.

Keynotes

I enjoyed all three keynotes. Mike Franklin's talk focused on AMPLab's research progress and the steady stream of artifacts they've assembled into the Berkeley Data Analytics Stack (BDAS -- pronounced "bad-ass" :)). In addition to Spark, the stack now includes SparkSQL, GraphX, MLLib, Spark Streaming, and Velox. All of these projects pose interesting systems questions and have made a lot of progress on the kinds of tooling that will come in handy for large scale data analysis beyond SQL queries.

Lada Adamic's talk was a fun visual tour through lots of interesting analyses of Facebook data that her team has been running. In particular, she's been using the data to understand what makes a particular piece of content rapidly gain a lot of popularity (or go "viral"). "Sharing" behavior is unique to social networks -- traditional web properties have spent lots of energy understanding what makes people click. The "share" action is a completely different beast to study. Lada reported that they have had some success predicting if a piece of content, once it reaches K users, if it will get to 2K. The prediction accuracy improves as we get to larger K. Unfortunately, she reported that the features that predict virality mostly have to do with the speed at which it is spreading, so we don't yet have a handle on how best to craft viral content

Thorsten Joachims had a technical keynote that followed the arc of his previous work on learning to rank in the context of search. He talked about careful design of interventions in interactive processes to elicit information that can be suitably leveraged by a machine learning algorithm to improve ranking. Lots of examples with evidence from experiments he ran with the arxiv search prototype that have since been reproduced by Yahoo, Baidu, etc.

Research Talks

Here's a subset of research talks that I thought were fun and interesting:

• Delayed-Dynamic-Selective (DDS) Prediction for Reducing Extreme Tail Latency in Web Search: a nice example of exploiting the nature of search servers to go beyond the standard techniques described in the Tail at Scale, which I talked about recently. The paper was the runner-up best paper award.
• Robust Tree-based Causal Inference for Complex Ad Effectiveness Analysis: applying causal inference techniques to ad-campaign data.
• FLAME: A Probabilistic Model Combining Aspect Based Opinion Mining and Collaborative Filtering: ideas on how to better model aspects of a user-rating when there's some text in addition to a numeric rating. Amazon product reviews being an obvious example.
• Just in Time Recommendations – Modeling the Dynamics of Boredom in Activity Streams: modeling repeat consumption of music and figuring out when a user wants to listen to more music of the same kind (same artist, same album) vs. when the user has gotten bored, and wants to move on to something else.
• User Modeling for a Personal Assistant: Srikant's talk on the user-modeling that goes into Google Now. Fun insights into the practical problems that were solved to build Google Now cards.
• Inverting a Steady-State: the best paper award winner from our group at Google.

The Tail at Scale

Another blog post recommending a paper to read ....

One of the challenges of building large systems on shared infrastructure like AWS and the Google Cloud is that you may have to think harder about dealing with variations in response time than you might if you were designing for dedicated hardware (like a traditional data warehouse).  This is obviously critical when you're building latency-critical services that power interactive experiences (websites, mobile apps, video/audio streaming services, etc.). Depending on the operating point, you're likely to see this effect even with batch processing systems like Map-Reduce and large scale machine learning systems.

A CACM article from Jeff Dean and Luiz Andre Barosso -- The Tail at Scale contains many valuable lessons on how to engineer systems to be robust to tail latencies. Some of the interesting ideas from the paper are to use:

1. Hedged requests: The client sends a request to one replica, and after a short delay, sends a second request to another replica. As soon as one response is received, the other request is cancelled.
2. Tied requests: The client sends a request to two replicas (also with a short delay) making sure the requests have metadata so that they know about each other. As soon as one replica starts processing a request, it sends a cancellation message to the other replica (this keeps the client out of the loop for the cancel-logic). Proper implementations of hedged requests and tied requests can significantly improve the 99th percentile latency with a negligible (2% - 3%) increase in the number of requests sent.
3. Oversharding/Micro-Partitions: The portion of data to be served is divided into many more shards than there are machines to serve them. Each server is assigned several shards to manage. This way, load balancing can be managed in a fairly fine-grained manner by moving a small number of shards at a time from one server to another. This allows us better control in managing the variability in load across machines. Systems like Bigtable (and open-source implementations like HBase), for instance, have servers that each manage dozens of shards.

There are lots of other ideas and interesting discussions in the paper. I strongly recommend it!

Machine Learning in Practice

A bunch of experienced Google engineers recently published a paper at a NIPS workshop titled "Machine Learning: The High Interest Credit Card of Technical Debt". This is a great paper that distills several dozen years worth of experience in building and maintaining a large-scale, complex, industrial-strength machine-learning system. If you haven't seen this paper I highly recommend it to get a sense for the patterns and anti-patterns they've observed along with supporting anecdotes.

The perspective is very different from a typical  ML paper -- you're not going to see a new model or a clever optimization strategy. The focus is mostly on the places where you have to go beyond the generally understood good software engineering practices to make the job of operating a large-scale ML system manageable. It's a short and fun read. Even if you've run an industrial ML system for a few years, you'll likely learn a trick or two from this paper.

Data management and Machine Learning -- Picking off Feature Engineering

Over the last few years, there have been many interesting efforts from the data management research community to support various kinds of machine learning workloads on large data sets. Projects like MLBase, MADLib, SystemML, ScalOps on Hyracks, as well as systems like Spark, Mahout, have surfaced many interesting ideas.

I think a recent line of work in the database community that focuses on the "feature engineering" phase of a machine learning project is particularly promising. Brainwash, outlined at CIDR in 2013 and Columbus, described at SIGMOD this year argue that the "feature engineering" pipeline that needs to be built to feed any particular learning algorithm tends to take up the majority effort in any machine learning project. In fact, I've heard many practitioners estimate the feature engineering and "pipeline work" to consume about 90% of the resources (time, people) in a machine learning project. This includes getting all the features together in one place, gathering statistics about the features, possibly normalizing them, and making sure any transformations applied correctly before handing it over to the learning algorithm.

Columbus  provides a library of functions in R to build up a feature selection pipeline. In return for expressing the feature selection workflow using the interface provided by this library, Columbus promises several optimizations by reasoning about 1) user-specified error tolerance, 2) sophistication of learning task, and 3) amount of reuse. In experiments, the naive execution strategy was more than an order of magnitude slower than the plan Columbus picked for the program. The optimizations exploit well-known opportunities -- view materialization from databases, sophisticated sampling of input data (constrained by target accuracy) to reduce overall work, and using QR factorization to solve repeated least-squares problems efficiently.

There are some (non-technical) flaws in the paper -- while some enterprise feature-engineering workloads may look like what they outline, things look very different in internet companies that repeatedly solve learning problems on large datasets. (Besides, R is often not the tool of choice for large learning problems, so this is a bit of a random nit.)  For ad-hoc machine learning tasks in enterprise settings, I can see this being a reasonable fit. Also with Columbus, are you still writing your feature-engineering code in R if you're using the library functions provided? You have to pass in selection predicated in strings parsed/interpreted by Columbus -- you're now limited not by what you can do in R, but what support is built into Columbus for feature engineering. Is this expressive enough to be broadly useful without having to hack up the R parser? (On a side-note, a DSL friendly language like Scala might offer some nice advantages here, but then you're no longer in an R console.)

I do think the choice of focusing on the feature-engineering workflow is brilliant, and perhaps the place where the data management community can add the most value. Broader "declarative framework for machine learning" approaches are riskier (even when they make the sensible choice of doing a domain specific language in a DSL-friendly host-language) and may take much longer to have an impact. I'm excited to see where this line of research leads.

I'd recommend reading the Columbus paper. It won SIGMOD's best paper, and is a fun read. There are several ideas here that might be broadly useful to the more ambitious systems that are trying to support full fledged machine learning workflows.

Recommendations: Why RMSE can be misleading

I've recently spent some time working on machine learning and recommendation engines. If you're building a recommendation engine, you're typically trying to optimize for some metric of "goodness" for the user. In a Netflix-like setting, it could be how much time does a user spend watching the content you recommended? Picking good offline metrics (without actually watching how the user is responding) can be really tricky. The RMSE (Root Mean Square Error), the staple of many research papers, can be particularly misleading in many cases.

Assume, we have 100 items that a user rates between 1 to 5 stars (much like in the Netflix problem). For simplicity, assume that the first three items have 5-star ratings, and the rest have a 1-star rating.

Product	True Rating	Algo. A Predictions	Algo. B Predictions
P001	5	2	1
P002	5	2	1
P003	5	2	1
P004	1	2	1
...	...	2	1
P100	1	2	1

Consider Algorithm A that predicts that all the ratings will be 2. The RMSE for this dataset = sqrt((97 + 27)/100) = 1.11. Now consider Algorithm B that predicts all ratings to be 1. The RMSE for this dataset is sqrt(48/100) = 0.693. Algorithm B produced a huge improvement in RMSE over algorithm A, but is it really any better at differentiating between the items that the user liked vs. ones that she didn't? If you are going to use the recommendations to solve a ranking problem, RMSE is a pretty useless measure in this context. A better metric would capture the fact that you're trying to use the recommendations to display the best items to the users, hoping that the user clicks/buys/watches/engages with/likes what you recommended. Being accurate on items way beyond the top few that the user is likely to engage with is not very useful at all.

Now on the other hand, if the "rating" we have is binary -- say someone "likes" the movie or not -- say 1 or 0. (In reality there's a third state, where someone watches a movie, and then doesn't rate it. You could can map this state to a 1 or 0 with a few application-specific assumptions). With a binary rating, the RMSE simply counts how many predictions you got right. Because what we really have here is a classification problem, and not a ranking problem, RMSE ends up being more reasonable.

There are several papers that talk about vastly superior metrics for ranking (that actually work in practice!) that I'll try and describe them in future posts.

Secondary Indexing in HBase

One of the problems I explored during the last few months at Almaden was secondary indexing in a scale-out NoSQL datastore. A solution I designed and implemented in collaboration with folks at IBM Watson has made it into the IBM BigInsights product and into EDBT 2014. Here's the full paper.

Secondary Indexes: Local and Global
There are two ways to think about secondary indexes in a distributed system -- "local indexes" and "global indexes". A local index simply indexes the data within each node. If you want to find all the rows which have attribute color set to 'red', each node in the system would be able to find the matching rows using a secondary index on the color column instead of doing a full scan. On the other hand, if you had a global secondary index on color, probing this would return all the keys which have color set to 'red'. This information is likely on just one node -- you don't have to query all the nodes in the system to locate the relevant keys.

Cassandra has has local secondary indexes for a while. HBase has had prototypes for both local and global indexes. See here for some discussion on possible implementations. Local indexes tend to be a little easier to design and implement since you don't have to deal with any notion of a distributed transaction. With a global index, you have to deal with the possibility that there might be a failure between updating the node with the base data and the node with the index data. Also, update latency for an indexed table gets much worse -- you need to update the base table, delete the old index entry, and update the index with the new entry. This translates to two updates and a read. For stores based on Bigtable-like tablets and a log-structured merge tree design, the read latency might even be worse than the write latency. Our paper describes a solution around both these problems for HBase : 1) guaranteeing atomicity in the face of failures, and 2) minimizing the impact to latency.

Asynchronous Indexing
The core novel idea in the first approach, which we call async-simple is to implement indexing as an asynchronous operation. Update the base table right away, and respond, but then guarantee that within a "short while", the indexes will be consistent. That is, you have read-your-writes consistency for base data, but a more relaxed form of consistency for the index data. For many applications this is a reasonable trade-off that gets you very good latency guarantees. For the applications that need better consistency guarantees, we proposed async-session, which provides session consistency for the index data. That is, you'll be able to see the effects of your own updates within a session. We show that both of these techniques can be implemented while guaranteeing atomic index updates with nearly no impact on update latency. You don't need distributed transactions to get this right. Here's how we solved this -- we added two new components to the RegionServer an Asynchronous Update Queue (AUQ) and an Asynchronous Processing Service (APS). The AUQ is an in-memory data structure that temporarily stores all base updates that require index processing. These live in the same RegionServer as the one responsible for the base row. The APS is a background service that de-queues from the AUQ, inserts new values into the index and deletes the corresponding old values if present.

Recovery is guaranteed by the fact that 1) the WAL entry for the base update has enough information to re-construct the index updates and re-populate the AUQ during recovery, 2) we use the same timestamp for the index updates as for the base data, and 3) we ensure that the AUQ can always be reconstructed using the base WAL entries without resorting to a separate log for the AUQ. Property 3) requires careful coordination between the AUQ and rolling forward the WAL to ensure that it is not garbage collected (when memtables get flushed to disk) until the APS has drained all the corresponding entries from the AUQ. Note that during recovery, we may redo some of the index updates that were done prior to failure. However, this is okay because re-applying an update with an identical timestamp is an idempotent operation. (See Pat Helland's paper here for a nice discussion on how to use idempotent messaging to avoid distributed transactions.)

Relaxed Consistency
This means there will be moments in time when the index is not consistent with the base data, and that may be visible to clients. This is unavoidable without an expensive distributed locking or a Generalized Snapshot Isolation-style solution. A small amount of client logic is usually enough to deal with this inconsistency. For applications that do not want to deal with this, we implemented session-consistency where you get to see all the updates made within your session, but this may not hold across sessions. We were able to do this with minor changes to the HBase driver -- you essentially track the necessary state in the client driver. This works fine for small sessions (few KB of in-session updates), which is likely to be the case for most applications that run on HBase style datastores.

The implementation relies mostly on using CoProcessors. In fact, async-simple required nearly no changes to the core, while async-session required changes to the client library. I think a similar implementation can be done in Cassandra as well. The experiments in the paper are pretty basic and straightforward. The most interesting result is that the latency of updates with async-simple and async-session is nearly the same as an update (Put operation) on an un-indexed table.
The main concern is that the throughput caps off lower than the un-indexed updates, but that's understandable that the indexed-update is 2 Puts + a read while the base update is simply 1 update. With some straighforward tuning of the sizes of the AUQ and memtable flush frequency, and assuming relatively cheap reads, we should be able to get indexed updates with the same latency as base updates maxing out at roughly a third the throughput of base updates.