Interesting Problems with Pandas Versions in INMOOSE

[stevenjbark]

I previously used the PyComBat library, but this has been migrated to INMOOSE. The problem I encountered is that Pandas versions are not compatible with INMOOSE.

Using Pandas=2.2.3, when importing
from inmoose.pycombat import pycombat_norm, pycombat_seq

I get a long list error culminating in ModuleNotFoundError: No module named 'pandas.core.index'.

I found a reference in StackOverflow indicating that the pandas.core.index was removed in Pandas>2.0.0, so I needed to downgrade. Once I downgraded to Pandas=1.5.3, this error was removed. HOWEVER, there is a new error when I tried to output a pandas dataframe using the .to_csv('filename') function: ModuleNotFoundError: No module named 'pandas.io.formats.csvs'.

So either I can use Pandas=1.5.3 to import the PyComBat functions without the ability to export the dataframes, or I can use Pandas=2.2.3 without the ability to use the PyComBat functions at all.

Please advise on this conundrum.

Best,
Steve Bark
Bioinformatics Analyst
Baylor College of Medicine

[EpigeneMax]

Dear Steven,

Thank you for reporting this.

We also use pandas 2.2.3, and we have not seen this error in our tests nor in our routine usage.
Could you please provide the full error message, so that we can investigate this issue further?

Best,

[liboxun]

I believe I encountered the same error as @stevenjbark

When I tried to run from inmoose.pycombat import pycombat_norm I got:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], [line 1](vscode-notebook-cell:?execution_count=2&line=1)
----> [1](vscode-notebook-cell:?execution_count=2&line=1) from inmoose.pycombat import pycombat_norm

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/__init__.py:6
      3 __version__ = importlib.metadata.version(__package__)
      5 from . import edgepy as edgepy
----> [6](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/__init__.py:6) from . import pycombat as pycombat
      7 from . import utils as utils
      8 from . import consensus_clustering as consensus_clustering

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/__init__.py:[1](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/__init__.py:1)
----> 1 from .pycombat_norm import pycombat_norm as pycombat_norm
      2 from .pycombat_seq import pycombat_seq as pycombat_seq

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/pycombat_norm.py:25
     22 import pandas as pd
     24 from ..utils.logging import LOGGER
---> [25](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/pycombat_norm.py:25) from .covariates import make_design_matrix
     27 # aprior and bprior are useful to compute "hyper-prior values"
     28 # -> prior parameters used to estimate the prior gamma distribution for multiplicative batch effect
     29 # aprior - calculates empirical hyper-prior values
     32 def compute_prior(prior, gamma_hat, mean_only):

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/covariates.py:20
     18 import numpy as np
     19 import pandas as pd
---> [20](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/inmoose/pycombat/covariates.py:20) from anndata import AnnData
     21 from patsy import DesignMatrix, dmatrix
     23 from ..utils import LOGGER, asfactor

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/anndata/__init__.py:[1](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/anndata/__init__.py:1)
----> 1 from .core.anndata import AnnData, Raw
      2 from .readwrite import (
      3     read_h5ad, read_loom, read_hdf,
      4     read_excel, read_umi_tools,
      5     read_csv, read_text, read_mtx,
      6     read_zarr,
      7 )
      8 from .readwrite import read_h5ad as read  # backwards compat / shortcut for default format

File ~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/anndata/core/anndata.py:[17](https://file+.vscode-resource.vscode-cdn.net/Users/bxli/Dropbox/3%20Resources/2%20Work/Gersbach%20lab/2_Analyses/Ana_Exp4/7.integrative_analysis/2.integrate_cpm_of_perturbations/~/anaconda3/envs/inmoose_env/lib/python3.12/site-packages/anndata/core/anndata.py:17)
     15 from numpy import ma
     16 import pandas as pd
---> 17 from pandas.core.index import RangeIndex
     18 from pandas.api.types import is_string_dtype, is_categorical
     19 from scipy import sparse

ModuleNotFoundError: No module named 'pandas.core.index'

With pandas 2.3.2 and inmoose 0.8.1