Skip to content

Error when importing python packages #38

Closed

Description

@PosoSAgapo

The python enviroment that I use in my rust package is anaconda python, however, error happened when I was tring to import pandas and many other packages

/opt/anaconda3/lib/python3.7/site-packages/numpy/__init__.py:140: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see https://github.com/IntelPython/mkl-service
  from . import _distributor_init
Traceback (most recent call last):
  File "src/main.rs", line 5, in <module>
    import pandas
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 17, in <module>
    "Unable to import required dependencies:\n" + "\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:
numpy: 

It shows I did not install numpy so that the inline-python could not import pandas, but the numpy is absolutely installed in my python directory.But the default package like sys works fine, then I use import sys to show the import directory of the python that I use in rust, the import directory is just the same as my anaconda python, which both are the same:

sys.path
['', '/opt/anaconda3/lib/python37.zip', '/opt/anaconda3/lib/python3.7', '/opt/anaconda3/lib/python3.7/lib-dynload', '/opt/anaconda3/lib/python3.7/site-packages', '/opt/anaconda3/lib/python3.7/site-packages/aeosa']

What could have possible caused this problem?

Activity

  1. de-vri-es commented on Aug 17, 2020

    @de-vri-es
    Contributor

    The error message you're showing starts with a warning from numpy, so python can find it.

    My guess is that while numpy calls it a warning, pandas or maybe something else is treating it like an error. Either way, it sounds like you should install the mkl-service package.

  2. PosoSAgapo commented on Aug 18, 2020

    @PosoSAgapo
    Author

    The error message you're showing starts with a warning from numpy, so python can find it.

    My guess is that while numpy calls it a warning, pandas or maybe something else is treating it like an error. Either way, it sounds like you should install the mkl-service package.

    I checked mkl-service and it was installed, the problem is that running the same python in terminal is working normally, but error only happens in the rust project,which is weird, because they are suppose to be the same python installed through anaconda. I also checked the sys.path to make sure the package import directory is same in both rust project and terminal python.

  3. de-vri-es commented on Aug 18, 2020

    @de-vri-es
    Contributor

    Try to import the mkl-service package directly and see what happens. Maybe numpy is hiding the error that occurs when importing it.

  4. PosoSAgapo commented on Aug 18, 2020

    @PosoSAgapo
    Author

    I tried directly import mkl, the error is different from import pandas, which shows the below error information:

    Traceback (most recent call last):
      File "src/main.rs", line 12, in <module>
        import mkl
      File "/opt/anaconda3/lib/python3.7/site-packages/mkl/__init__.py", line 48, in <module>
        with RTLD_for_MKL():
      File "/opt/anaconda3/lib/python3.7/site-packages/mkl/__init__.py", line 33, in __enter__
        import ctypes
      File "/opt/anaconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: dlopen(/opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Symbol not found: _PyByteArray_Type
      Referenced from: /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
      Expected in: flat namespace
     in /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
    thread 'main' panicked at 'python!{...} failed to execute', /Users/chenbowen/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13:23
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    It directly shows a import error and did not find the _PyByteArray_Type, but the same python works fine when importing the same package

  5. m-ou-se commented on Aug 18, 2020

    @m-ou-se
    Owner

    Oh, interesting. Looks like some dynamic linker stuff is going wrong on Mac. I don't have a Mac to test things on though. I assume import ctypes gives the same error for you?

    fn main() {
        python! {
            import ctypes;
        }
    }
  6. m-ou-se commented on Aug 18, 2020

    @m-ou-se
    Owner

    What's the output of otool -L target/debug/yourprogram?

  7. m-ou-se commented on Aug 18, 2020

    @m-ou-se
    Owner

    These look related:

    It looks like this is not an issue with iniline-python, but with PyO3 or your Python installation. To make sure, you can check if importing the module directly with pyo3 works:

    fn main() {
    	pyo3::Python::acquire_gil().python().import("ctypes").unwrap();
    }

    If that also fails, please open an issue with PyO3.

  8. PosoSAgapo commented on Aug 19, 2020

    @PosoSAgapo
    Author

    Oh, interesting. Looks like some dynamic linker stuff is going wrong on Mac. I don't have a Mac to test things on though. I assume import ctypes gives the same error for you?

    fn main() {
        python! {
            import ctypes;
        }
    }

    Importing Ctype also cuasing an error information:

     File "src/main.rs", line 12, in <module>
        import ctypes;
      File "/opt/anaconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: dlopen(/opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Symbol not found: _PyByteArray_Type
      Referenced from: /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
      Expected in: flat namespace
     in /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
    thread 'main' panicked at 'python!{...} failed to execute', /Users/chenbowen/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13:23
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    And directly importing it from Pyo3 also casuing a problem:

     thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x10a947920, PhantomData) }', src/main.rs:14:59
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    It seems this might be a problem of PyO3 or my Python installation, but why the python that runs in the rust project does not behave the same way like the terminal python? They are the same python that I checked using sys.version and also the import directory is also the same which I checked using sys.path

  9. PosoSAgapo commented on Aug 19, 2020

    @PosoSAgapo
    Author

    otool -L target/debug/yourprogram

    Running this command, the output is like below:

    target/debug/ruststudy:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)

  10. m-ou-se commented on Aug 19, 2020

    @m-ou-se
    Owner

    And directly importing it from Pyo3 also casuing a problem

    In that case, please open an issue with the PyO3 project. If it isn't working in PyO3, there's nothing the inline-python crate can do. (Make sure to add a link to this issue for context. :) )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions