FAQ

• What does “compactem” mean?

  Think of “compactem” as “compact M” (“M” for models), or “compact ‘em”. Also, I later discovered that the word means “compact” in Catalan. That had to be a sign.

• Can I use this on Deep Learning models too?

  In theory yes. But I would NOT recommend it in practice since multiple model refits are performed to obtain a compact model; this is a massive compute cost for Deep Learning models. It would be more practical to look at techniques like pruning or distillation as they are specialized for Deep Learning.

• So, if my small model is a logistic regression (LR) classifier, and the oracle is a highly accurate neural network (NN), does my LR become as accurate as the NN?

  Nope, the LR model might become a better LR model. But, its still a LR model and is a linear classifier.

• Where can I dive into details about the algorithm?

  Here is our paper with all the details.

• Is this another form of the Minimum Description Length (MDL)?

  No. MDL finds the smallest model among models that accurately classify the data. Here, we find the most accurate model given a model size.

• What are your plans for this package?

  I want to keep experimenting with different aspects of the problem, like exploring other optimization backends, or distributions to represent the search space, etc. Successful approaches would be eventually integrated into the package. However, I have no planned timeline for this.