Skip to content

numpy recipe bug arm64-v8a #1873

Closed

Description

@musarra

Versions

  • Python: 3
  • OS: linux
  • Kivy: 1.11
  • Cython: not sure

Description

// REPLACE ME: What are you trying to get done, what has happened, what went wrong, and what did you expect?

Google Play does not support 32 bit libraries and the armeabi-v7a cannot be used. This means the apk I created and worked for months could not be uploaded. I tried to fix that issue changing the architecture to 64 bit arm64-v8a. Problem is that I have numpy in requirements and it has an issue with the recipe. The apk get's built but crashes immediately, it never opens.

I am not expert programmer but my suggestions to solve/check the issue looking to numpy recipe are:

  1. this url = 'https://pypi.python.org/packages/source.....etc' does not exist. I would replace for https://pypi.org/simple/numpy/numpy-1.16.4.zip instead
  2. in recipe, version = '1.15.1'. In numpy website, version = 1.16.4

I built a very simple HelloWorld app in kivy to demonstrate this issue below.

Command:

buildozer android debug

Logs

06/18 23:34:21: Launching hwtest-0.1-debug
$ adb shell am start -n "org.test.hwtest/org.kivy.android.PythonActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER
Client not ready yet..Waiting for process to come online
Waiting for process to come online
Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page.
I/python: [INFO ] [Factory ] 184 symbols loaded
I/python: [INFO ] [Image ] Providers: img_tex, img_dds, img_sdl2, img_gif (img_pil, img_ffpyplayer ignored)
I/python: Traceback (most recent call last):
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/core/init.py", line 16, in
I/python: ImportError: dlopen failed: cannot locate symbol "__fixunstfsi" referenced by "/data/data/org.test.hwtest/files/app/_python_bundle/site-packages/numpy/core/multiarray.so"...

 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/app/main.py", line 3, in <module>

I/python: File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/init.py", line 142, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/add_newdocs.py", line 13, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/lib/init.py", line 8, in
I/python: File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/lib/type_check.py", line 11, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/core/init.py", line 26, in
ImportError:
I/python: Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.

 Original error was: dlopen failed: cannot locate symbol "__fixunstfsi" referenced by "/data/data/org.test.hwtest/files/app/_python_bundle/site-packages/numpy/core/multiarray.so"...
 
Python for android ended.

W/IPCThreadState: Calling IPCThreadState::self() during shutdown is dangerous, expect a crash.
W/org.test.hwtes: 0xebadde09 skipped times: 0
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x184 in tid 26345 (org.test.hwtest), pid 26345 (org.test.hwtest)
Application terminated.

Activity

  1. RobertFlatt commented on Jun 18, 2019

    @RobertFlatt
    Contributor

    Saw same issue with 64 bit (but not 32 bit) numpy build.

    The numpy release notes https://github.com/numpy/numpy/releases contain the following under release 1.16.0

    Highlight
    Improved support for the ARM

    and later

    ARM support updated
    Support for ARM CPUs has been updated to accommodate 32 and 64 bit targets,
    and also big and little endian byte ordering. AARCH32 memory alignment issues
    have been addressed. CI testing has been expanded to include AARCH64 targets
    via the services of shippable.com.

    This might be saying ARM 64 bit support added? Perhaps the issue is related to defaulting to 1.15.1 ?

    Tried requirements numpy==1.16.0 but this gives a build (fix-numpy.patch) error.

  2. musarra commented on Jun 19, 2019

    @musarra
    Author
  3. inclement commented on Jun 20, 2019

    @inclement
    Member

    @Ham-Merhead Thanks for the note, it looks like updating to numpy 1.16.0 is probably the obvious thing worth trying.

  4. RobertFlatt commented on Jun 20, 2019

    @RobertFlatt
    Contributor

    @inclement
    Been trying to build numpy 1.16.4 using the p4a-recipes directory
    Issues with fix-numpy.patch , and python-fixes.patch , other patches are OK

  5. musarra commented on Jun 20, 2019

    @musarra
    Author
  6. inclement commented on Jun 20, 2019

    @inclement
    Member

    @musarra You'll need to try what @Ham-Merhead mentioned, updating the patches as necessary for numpy 1.16. This isn't that hard, but may be tricky to get into if you aren't familiar with it.

    I'll try to look at this at the weekend, I actually have a 64 bit phone now to test on.

  7. musarra commented on Jun 20, 2019

    @musarra
    Author
  8. musarra commented on Jun 21, 2019

    @musarra
    Author

    By the way guys, I think is not worth it to open another bug tag, but in my simple HelloWorld android app, if I set py file to 'import matplotlib.pyplot as plt' and buildozer.spec to 'requirements = python3,kivy,matplotlib' it does not build the apk and the error message is below. Guess what? is related to numpy....

    [INFO]:    Building matplotlib for arm64-v8a
    [INFO]:    matplotlib apparently isn't already in site-packages
    [INFO]:    Building compiled components in matplotlib
    [INFO]:    -> directory context /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/matplotlib-python3/arm64-v8a__ndk_target_21/matplotlib
    [INFO]:    -> running python setup.py build_ext -v
               Exception in thread background thread for pid 9686:                 
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 1540, in wrap
        fn(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 2459, in background_thread
        handle_exit_code(exit_code)
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 2157, in fn
        return self.command.handle_command_exit_code(exit_code)
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 815, in handle_command_exit_code
        raise exc
    ErrorReturnCode_1: 
    
      RAN: /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python setup.py build_ext -v
    
      STDOUT:
    ============================================================================
    Edit setup.cfg to change the build options
    
    BUILDING MATPLOTLIB
                matplotlib: yes [3.0.3]
                    python: yes [3.7.1 (default, Jun 21 2019, 19:47:23)  [GCC
                            7.4.0]]
                  platform: yes [linux]
    
    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [not found. pip may install it below.]
          install_requires: yes [handled by setuptools]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.8.1]
                       png: yes [version 1.6.34]
                     qhull: yes [pkg-config information for 'libqhull' could not
                            be found. Using local copy.]
    
    OPTIONAL SUBPACKAGES
               sample_data: no  [skipping due to configuration]
                  toolkits: yes [installing]
                     tests: no  [skipping due to configuration]
            toolkits_tests: no  [skipping due to configuration]
    
    OPTIONAL BACKEND EXTENSIONS
                       agg: yes [installing]
                     tkagg: no  [Disabled by patching during Android build]
                    macosx: no  [skipping due to configuration]
                 windowing: no  [skipping due to configuration]
    
    OPTIONAL PACKAGE DATA
                      dlls: no  [skipping due to configuration]
    
    **Download error on https://pypi.org/simple/numpy/: unknown url type: https -- Some packages may not be found!**
    **Couldn't find index page for 'numpy' (maybe misspelled?)**
    **Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!**
    **No local packages or working download links found for numpy==1.15.1**
    Traceback (most recent call last):
      File "setup.py", line 297, in <module>
        cmdclass=cmdclass,
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy==1.15.1')
    
    
      STDERR:
    
    
    Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 1186, in <module>
        main()
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 1180, in main
        ToolchainCL()
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 663, in __init__
        getattr(self, args.subparser_name.replace('-', '_'))(args)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 152, in wrapper_func
        build_dist_from_args(ctx, dist, args)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 205, in build_dist_from_args
        args, "ignore_setup_py", False
      File "pythonforandroid/build.py", line 555, in build_recipes
      File "pythonforandroid/recipe.py", line 883, in build_arch
      File "pythonforandroid/recipe.py", line 938, in build_compiled_components
      File "pythonforandroid/recipe.py", line 895, in build_compiled_components
      File "pythonforandroid/logger.py", line 178, in shprint
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 863, in next
        self.wait()
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 792, in wait
        self.handle_command_exit_code(exit_code)
      File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 815, in handle_command_exit_code
        raise exc
    sh.ErrorReturnCode_1: 
    
      RAN: /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python setup.py build_ext -v
    
      STDOUT:
    ============================================================================
    Edit setup.cfg to change the build options
    
    BUILDING MATPLOTLIB
                matplotlib: yes [3.0.3]
                    python: yes [3.7.1 (default, Jun 21 2019, 19:47:23)  [GCC
                            7.4.0]]
                  platform: yes [linux]
    
    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [not found. pip may install it below.]
          install_requires: yes [handled by setuptools]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.8.1]
                       png: yes [version 1.6.34]
                     qhull: yes [pkg-config information for 'libqhull' could not
                            be found. Using local copy.]
    
    OPTIONAL SUBPACKAGES
               sample_data: no  [skipping due to configuration]
                  toolkits: yes [installing]
                     tests: no  [skipping due to configuration]
            toolkits_tests: no  [skipping due to configuration]
    
    OPTIONAL BACKEND EXTENSIONS
                       agg: yes [installing]
                     tkagg: no  [Disabled by patching during Android build]
                    macosx: no  [skipping due to configuration]
                 windowing: no  [skipping due to configuration]
    
    OPTIONAL PACKAGE DATA
                      dlls: no  [skipping due to configuration]
    
    Download error on https://pypi.org/simple/numpy/: unknown url type: https -- Some packages may not be found!
    Couldn't find index page for 'numpy' (maybe misspelled?)
    Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!
    No local packages or working download links found for numpy==1.15.1
    Traceback (most recent call last):
      File "setup.py", line 297, in <module>
        cmdclass=cmdclass,
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy==1.15.1')
    
  9. RobertFlatt commented on Jun 22, 2019

    @RobertFlatt
    Contributor

    Some progress, but still stuff to figure out:

    For 1.16.4 in the patches list I

    1. commented fix-numpy.patch -- because I have no understanding of the test methodology
    2. built a new python-fixes.patch (attached) -- no judgement here, just doing the same thing as before
      python-fixes.zip

    However the build fails
    error: Command "aarch64-linux-android-ld ............... failed with exit status 127

    Don't know why aarch64-linux-android-ld is not found, except that I cant find it in python-for-android/build .

    @musarra This works for p4a , Buildozer I don't know about:

    1. Create a directory "p4a-recipes" in your project directory
    2. copy p4a numpy recipe there from github
    3. replace python-fixes.patch with the one attached
    4. Edit __init__.py
      a) change version
      b) comment line containing 'fix-numpy.patch'
      c) change 'patches' in the following 4 lines to an absolute path e.g. '/usr/me/stuff/patches' (or whatever)

    p4a will build numpy from these local instructions in place of the default instructions. (To check it is really building from there, insert a typo so the build fails!)

    See if you can make some progress....

  10. musarra commented on Jun 22, 2019

    @musarra
    Author
  11. musarra commented on Jun 24, 2019

    @musarra
    Author
  12. inclement commented on Jun 24, 2019

    @inclement
    Member

    @opacam Mentioned that it worked in a 64 bit build with ndk 19, in #1722

  13. musarra commented on Jun 24, 2019

    @musarra
    Author
  14. RobertFlatt commented on Jun 25, 2019

    @RobertFlatt
    Contributor

    @musarra
    I don't think Buildozer supports 19 yet (read the comments further up #1722 ).
    I tried 19c with p4a (64 bit target), I get this message:

    [INFO]:    Found NDK revision 19.2.5345600
    [WARNING]: Maximum recommended NDK version is 17c
    [WARNING]: Newer NDKs may not be fully supported by p4a.
    

    And after a while it fails (as expected):
    [ERROR]: Build failed: Couldn't find executable for CC. This indicates a problem locating the aarch64-linux-android-gcc executable in the Android NDK, not that you don't have a normal compiler installed. Exiting.

    aarch64-linux-android-gcc is not part of 19c , and p4a/Bulldozer tool flow changes (in this case to clang) are required for 19. So I would say for the moment we cant get there from here (via NDK 19). However NDK 19 looks like it will be a future route.

  15. musarra commented on Jun 26, 2019

    @musarra
    Author
  16. 10 remaining items

  17. inclement commented on Jun 29, 2019

    @inclement
    Member

    @musarra This won't be in the release branch that buildozer uses for a while, you need to use p4a master or set the p4a.branch in buildozer.spec to develop. If you already did that, please post the log.

  18. musarra commented on Jun 29, 2019

    @musarra
    Author
  19. inclement commented on Jun 29, 2019

    @inclement
    Member

    The matplotlib issue is probably environment related somehow, openssl is probably missing in the local python build. I'm not sure about the details though, I don't know what's different about my local env (which works okay). There's another issue open to track this.

  20. musarra commented on Jun 29, 2019

    @musarra
    Author
  21. AndreMiras commented on Jun 29, 2019

    @AndreMiras
    Member

    No it's definitely what @inclement said. The system doesn't have socket or other lib build with ssl support hence https schema not supported

  22. musarra commented on Jun 29, 2019

    @musarra
    Author
  23. AndreMiras commented on Jun 29, 2019

    @AndreMiras
    Member

    I would suggest to make sure your system libs are not broken https://stackoverflow.com/questions/28376506/urllib-https-request-urlopen-error-unknown-url-type-https
    This comes outside p4a to me. By the way it seems like your run p4a from python2, plus some of the deps (e.g. sh) were installed via sudo pip install which I would not recommend. I would recommend using pip install --user instead, that way it's easy to wipe out everything if you ever mess with your deps

  24. musarra commented on Jun 29, 2019

    @musarra
    Author
  25. musarra commented on Jun 30, 2019

    @musarra
    Author
  26. AndreMiras commented on Jun 30, 2019

    @AndreMiras
    Member

    Glad you could solve it only installing some system deps.
    As for how to use matplotlib, I've no clue, but you should check that from their website or tutorial on the web.
    Let's migrate support conversation to Discord then. Good luck!

  27. musarra commented on Jun 30, 2019

    @musarra
    Author
  28. AndreMiras commented on Jun 30, 2019

    @AndreMiras
    Member

    I see that makes sense. Well I've never played with it, but I'm sure you should find someone on Discord.

  29. Fabiocke commented on Nov 14, 2019

    @Fabiocke

    I solve the problem and it's very simple, just add: numpy == 1.16.4
    in your buildozer.spec requiriments

  30. inclement commented on Nov 14, 2019

    @inclement
    Member

    numpy 1.16.4 is the current default, as of a few months ago.

  31. GaspHard commented on May 25, 2020

    @GaspHard

    This does not work for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions