Collapse multiple submodules to one Cython extension

by

First off, I should note that it’s impossible to compile a single .so file with sub packages using Cython. So if you want sub packages, you’re going to have to generate multiple .so files, as each .so can only represent a single module.

Second, it doesn’t appear that you can compile multiple Cython/Python files (I’m using the Cython language specifically) and link them into a single module at all.

I’ve tried to compile multiply Cython files into a single .so every which way, both with distutils and with manual compilation, and it always fails to import at runtime.

It seems that it’s fine to link a compiled Cython file with other libraries, or even other C files, but something goes wrong when linking together two compiled Cython files, and the result isn’t a proper Python extension.

The only solution I can see is to compile everything as a single Cython file. In my case, I’ve edited my setup.py to generate a single .pyx file which in turn includes every .pyx file in my source directory:

includesContents = ""
for f in os.listdir("src-dir"):
    if f.endswith(".pyx"):
        includesContents += "include \"" + f + "\"\n"

includesFile = open("src/extension-name.pyx", "w")
includesFile.write(includesContents)
includesFile.close()

Then I just compile extension-name.pyx. Of course this breaks incremental and parallel compilation, and you could end up with extra naming conflicts since everything gets pasted into the same file. On the bright side, you don’t have to write any .pyd files.

I certainly wouldn’t call this a preferable build method, but if everything absolutely has to be in one extension module, this is the only way I can see to do it.

More Related Contents:

  1. Compiling with cython and mingw produces gcc: error: unrecognized command line option ‘-mno-cygwin’
  2. How to tell distutils to use gcc?
  3. Make distutils look for numpy header files in the correct place
  4. Using Cython To Link Python To A Shared Library
  5. How do I import a module given the full path?
  6. Unable to import a module that is definitely installed
  7. Make executable file from multiple pyx files using cython
  8. How to use MinGW’s gcc compiler when installing Python package using Pip?
  9. What is the recommended way of allocating memory for a typed memory view?
  10. What do I need to read Microsoft Access databases using Python?
  11. MANIFEST.in ignored on “python setup.py install” – no data files installed?
  12. How do I import other Python files?
  13. calling dot products and linear algebra operations in Cython?
  14. How to get filename of the __main__ module in Python?
  15. DistutilsOptionError: must supply either home or prefix/exec-prefix — not both
  16. How can I add post-install scripts to easy_install / setuptools / distutils?
  17. Python: ‘Private’ module in a package
  18. Filtering (reducing) a NumPy Array
  19. How do I extend a python module? Adding new functionality to the `python-twitter` package
  20. Fortran – Cython Workflow
  21. Is there a standard way to create Debian packages for distributing Python programs?
  22. Difference between np.int, np.int_, int, and np.int_t in cython?
  23. Definition of def, cdef and cpdef in cython
  24. cython shared memory in cython.parallel.prange – block
  25. ImportError: libSM.so.6: cannot open shared object file: No such file or directory
  26. Import python module NOT on path
  27. Are executables produced with Cython really free of the source code?
  28. Fast string array – Cython
  29. Extending python – to swig, not to swig or Cython
  30. Call python code from c via cython

Leave a Comment