[New Model] Add TabLDM foundation model - #487
Conversation
|
Heyho, very cool, thank you for your contribution! Can you report back once you have benchmark results for your method? I will then work on integrating the model and confirming your results. Moreover, I suggest not vendorizing your own model. This is mostly a workaround for when we don't control the model's codebase. Here, please try to keep your model and interface in their own GitHub codebases controlled by you, make them pip-installable, and then install/import them in the PR. Also, do you have a blog post or documentation on your method? |
LennartPurucker
left a comment
LennartPurucker
left a comment
There was a problem hiding this comment.
very clean otherwise, cool!
| # Vendored under `_vendor/` (not on PyPI). Most runtime deps (torch, numpy, scikit-learn, | ||
| # scipy, psutil, tqdm, huggingface_hub) are already in TabArena's base tree, but `einops` | ||
| # (used by `_model/rope.py`) is not, so it is the one real extra dependency. | ||
| pip_extra=("einops",), |
There was a problem hiding this comment.
You can add the install of your own package here
There was a problem hiding this comment.
I completely agree. I am currently removing the _vendor/ directory. I have set up the standalone inference package in my own repository and will add it to pyproject.toml via "git+https://github.com/xiaomi-research/xiaomi-tabldm.git@3090d4f3da420e25a482bead97c9fe061607bffb" to ensure reproducibility.
| self.model = model_cls(device=device, n_jobs=num_cpus, **hps) | ||
| self.model.fit(X, y) | ||
|
|
||
| def _predict_proba(self, X: pd.DataFrame, **kwargs) -> np.ndarray: |
There was a problem hiding this comment.
I am not sure you need this function, the default wrapper should to about the same
There was a problem hiding this comment.
You are absolutely right. I was over-engineering the _predict_proba(). I have removed the custom overrides and will rely entirely on the default AbstractTorchModel implementation to keep the wrapper clean.
|
Update since opening this PR — three things changed upstream/environment-side, plus a local validation run: 1. Switched from vendored code to a real pip dependency. 2. Renamed the model end-to-end: 3. The HF checkpoint repo is no longer gated. 4. Ran the TabArena-v0.1 "lite" subset locally, end to end.
Full TabArena-v0.1 (816-task) cluster run + upload via the |
|
@LennartPurucker Apologies for the extra commit here. The upstream Xiaomi-TabLDM HF checkpoint(occams/Xiaomi-TabLDM) was updated with 4 new MoE hyperparameters that the previously pinned commit didn't support, so loading the classifier checkpoint failed with a TypeError. This bumps the pin to the upstream fix commit (6773a30) that adds support for these parameters, keeping the wrapper aligned with the latest published checkpoint. Verified both classifier and regressor checkpoints load successfully after the bump. |
|
Very cool, thank you for your effort @occamsX! I will add this to my TODO list for next week to run the model and get it on the leaderboard by the 13th of September. |
2 participants
Adds TabLDM: a ~70M-parameter in-context-learning tabular foundation model with a dual-stream column embedder and a sparse Mixture-of-Experts (MoE1) backbone, trained on large-scale synthetic tabular data. Like TabPFN,
fitdoes not update weights; it only preprocesses the context and loads the pretrained checkpoint, and prediction runs through a single forward pass.Wrapper: models/tabldm/model.py
Codebase: https://github.com/xiaomi-research/xiaomi-tabldm
Checkpoints: https://huggingface.co/occams/Xiaomi-TabLDM
Technical report: not released yet (no paper/BibTeX published upstream).
has_raw/has_processed/has_resultsare allFalseandverified=Falseininfo.py— no benchmark run exists yet.Local fit is now verified for
binary: a CPU-forced smoke fit (ag_args_fit={"num_gpus": 0}) ran_fit's full path end to end (resource negotiation, preprocessing,model_cls(...).fit(X, y)) for both a single-fold fit + refit and a 2-fold bagged fit (sequential_localfold-fitting) + refit, each completing with a real validation accuracy and no errors. Reaching this point needed two environment fixes, unrelated to the wrapper code: the sandbox sits behind a corporate TLS-intercepting proxy whose CA isn't in the venv's bundledcertifistore (fixed by pointingREQUESTS_CA_BUNDLE/SSL_CERT_FILEat the system bundle, which already trusts it), andoccams/Xiaomi-TabLDMis a gated HF repo ("gated": "auto"), so the checkpoint download needs an HF token from an account that requested and was granted access. TODO(user): confirm benchmark/CI nodes will have such a token available (e.g.,HUGGING_FACE_HUB_TOKEN), since without one the checkpoint download returns 403s regardless of network setup.Changes
Notes
fitignoresX_val/y_val/time_limit(in-context-learning model, no training loop); preprocessing is handled by the vendored estimator itself.refit_folds=True+sequential_localfold-fitting (matching TabICL/TabSwift/LimiX): refitting one model on all data gives faster inference at similar quality to a bagged ensemble, and sequential fitting avoids fold contention on the shared HF checkpoint cache.can_hpo=False) and nowarmup()override yet —AbstractTorchModelcovers generic torch/CUDA warmup;prefetch_weights()already pre-downloads both checkpoints ahead of the timed fit. TODO(user): decide whether checkpoint prefetch/load should also happen during warmup once a run is possible.tests/tabarena/models/smoke_configs.pyoverride yet. TODO(user): once the checkpoint download is unblocked and a toy fit is verified, decide whether the default config needs a lighter override.date="2026-08-31"ininfo.pyis a placeholder (today's date, not a run date).verified=Falseand nosuite/cache_kwargsyet — filled in by the upload flow once a benchmark run exists.