PyBoy•2mo ago

Brainstorming: adapting lcd.py to android

Hi everyone!😁 I've made some progress adapting pyboy to android using kivy/buildozer. It works perfectly on some games, but unfortunately not on the one im interested, which is pokemon crystal. I get consistent crashes on the same frames everytime. For example, on the loading screen when suicune is running and you have to press start. The logs point to pyboy/core/lcd.so everytime, probably because it is trying to access memory which doesnt exist. It is really difficult to debug because i havent found the way to make a print which i can see while running on android. It doesnt crash on desktop. I feel really stuck on this, i was hoping you could give me some ideas to either fix it (i can addapt lcd.py) or either debugging it. Every bit of help is welcome! ❤️ Here are the crashing logs: 07-20 15:33:15.749 18477 18477 F DEBUG : NOTE: /data/data/org.test.myapp/files/ app/_python_bundle/site-packages/kivy/_clock.so 07-20 15:33:15.749 18477 18477 F DEBUG : NOTE: /data/data/org.test.myapp/files/ app/_python_bundle/site-packages/numpy/random/bit_generator.so 07-20 15:33:15.749 18477 18477 F DEBUG : NOTE: /data/data/org.test.myapp/files/ app/_python_bundle/site-packages/pyboy/core/lcd.so 07-20 15:33:15.749 18477 18477 F DEBUG : NOTE: /data/data/org.test.myapp/files/ app/_python_bundle/site-packages/pyboy/core/mb.so 07-20 15:33:15.749 18477 18477 F DEBUG : NOTE: /data/data/org.test.myapp/files/ app/_python_bundle/site-packages/pyboy/pyboy.so 07-20 15:33:15.749 18477 18477 F DEBUG : #00 pc 000000000001c9ac /data/data/ org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so 07-20 15:33:15.749 18477 18477 F DEBUG : #01 pc 000000000001c228 /data/data/ org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so 07-20 15:33:15.749 18477 18477 F DEBUG : # 07-20 ... And more, but im on the word limit already 😆

45 Replies

sergiomelero4OP•2mo ago

More info: It crashes on tick() It doesnt crash if render=False

Sky•2mo ago

Have you tried adding a bootrom? I would also see if for some reason, you get a similar error on gold, which is also playable with cgb = False

Bækalfen•2mo ago

Hmm. It's really strange that it doesn't fail immediately if so Can you try this? https://github.com/mattcurrie/cgb-acid2/releases/download/v1.1/cgb-acid2.gbc

Bækalfen•2mo ago

It's just a graphics test ROM https://github.com/mattcurrie/cgb-acid2

GitHub

GitHub - mattcurrie/cgb-acid2: 😀 The Acid2 test, now for Game Bo...

😀 The Acid2 test, now for Game Boy Color! 😀. Contribute to mattcurrie/cgb-acid2 development by creating an account on GitHub.

Bækalfen•2mo ago

It would be really helpful if you can post the whole stacktrace. Doesn't matter if it's long Did you compile for 64-bit?

sergiomelero4OP•2mo ago

Hi again guys! Sorry for the delay. I tried the Acid2 test and it crashes imediatelly after the pyboy intro. The logs are mostly the same as in pokemon crystal: #00 pc 000000000001c9d8 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so #01 pc 000000000001c23c /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so #02 pc 000000000001866c /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so #03 pc 0000000000016c48 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/mb.so #04 pc 0000000000033e64 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/pyboy.so #05 pc 00000000000344bc /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/pyboy.so #06 pc 00000000000475d8 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/pyboy.so #07 pc 0000000000012310 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/numpy/random/bit_generator.so #08 pc 00000000001f4550 /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (_PyObject_MakeTpCall+328) #09 pc 00000000002cc12c /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (_PyEval_EvalFrameDefault+17704) #10 pc 00000000002c7b64 /data/app/zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so #11 pc 000000000001cffc /data/data/org.test.myapp/files/app/_python_bundle/site-packages/kivy/_clock.so #12 pc 0000000000022e18 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/kivy/_clock.so #13 pc 0000000000034680 /data/data/org.test.myapp/files/app/_python_bundle/site-packages/kivy/_clock.so #14 pc 00000000001f49e8 /data/app/zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (PyObject_Vectorcall+72) #15 pc 00000000002cc12c /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (_PyEval_EvalFrameDefault+17704) #16 pc 00000000002c7a34 /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (PyEval_EvalCode+268) #17 pc 0000000000318938 /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (_PyRun_SimpleFileObject+1188) #18 pc 00000000003191fc /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libpython3.11.so (PyRun_SimpleFileExFlags+60) #19 pc 0000000000002740 /data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libmain.so (SDL_main+2664) (BuildId: 530c351246f588e0e63a15aec80493b9a9dd3d0f) #20 pc 00000000000b10f4 / data/app/~~zi0lZx0kJb2miHMfChB7EQ==/org.test.myapp-3gA0qvgT1HaYFz4CwOB5_w==/lib/arm64/libSDL2.so (Java_org_libsdl_app_SDLActivity_nativeRunMain+772) (BuildId: d1fc1a1b5aeed6101dd03188f643f3403d18d80e) #21 pc 000000000037ef70 /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #22 pc 0000000000368a40 /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #23 pc 0000000000361e5c /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod, art::Thread, art::ShadowFrame&, art::Instruction const, unsigned short, bool, art::JValue)+2048) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #24 pc 0000000000770ba4 /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext)+12208) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #25 pc 00000000003815d8 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #26 pc 000000000000cddc <anonymous:7d5a17e000> (org.libsdl.app.SDLMain.run+0) #27 pc 00000000003535dc /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+1932) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #28 pc 000000000037f098 /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #29 pc 0000000002004568 /memfd:jit-cache (deleted) (offset 0x2000000) (java.lang.Thread.run+136) #30 pc 0000000000368774 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #31 pc 0 000000000353f24 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread, unsigned int, unsigned int, art::JValue, char const)+132) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #32 pc 0000000000947748 /apex/com.android.art/lib64/libart.so (art::detail::ShortyTraits<(char)86>::Type art::ArtMethod::InvokeInstance<(char)86>(art::Thread, art::ObjPtr<art:🪞:Object>, art::detail::ShortyTraits<>::Type...)+60) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #33 pc 0000000000636e80 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void)+1344) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #34 pc 0000000000636930 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallbackWithUffdGc(void)+8) (BuildId: 4ccb65ae9ac5ad5da3af5a342d5b0b92) #35 pc 0000000000070578 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_start(void*)+200) (BuildId: bc5c61ed852b3f4266cdfdd25499ef95) #36 pc 0000000000061910 /apex/com.android.runtime/lib64/bionic/libc.so (start_thread+64) (BuildId: bc5c61ed852b3f4266cdfdd25499ef95

Bækalfen•2mo ago

Can you send the log as a file?

sergiomelero4OP•2mo ago

Oh sorry, of course

Bækalfen•2mo ago

Can you try the black/white version while you’re at it? https://github.com/mattcurrie/dmg-acid2

GitHub

GitHub - mattcurrie/dmg-acid2: 😀 The Acid2 test, now for the ori...

😀 The Acid2 test, now for the original Game Boy! 😀 - GitHub - mattcurrie/dmg-acid2: 😀 The Acid2 test, now for the original Game Boy! 😀

Bækalfen•2mo ago

Then we can maybe find out if it’s in the color code or the general code Also, what parameters are you supplying PyBoy(…)?

sergiomelero4OP•2mo ago

Here is the extended version with adb-logcat, ibelieve the crash starts with /data/data/org.test.myapp/files/app/_python_bundle/site-packages/pyboy/core/lcd.so

pyboyacid2crash-logs...

sergiomelero4OP•2mo ago

And it is compiled for arm64 v8a This is my pyboy instance: self.pyboy = PyBoy(self.rom_path, window="null", sound_emulated=True, sound_volume=100)

sergiomelero4OP•2mo ago

Black and wiki version working just fine! 😁

Bækalfen•2mo ago

How about dmg-acid2, but set cgb=True: PyBoy(self.rom_path, window="null", sound_emulated=True, sound_volume=100, cgb=True)?

sergiomelero4OP•2mo ago

Still crashes on cgb=True 😦

pyboycrystalcrash-lo...

Bækalfen•2mo ago

Ok, I kind of expected that. So it’s narrowing down to the cgb code in LCD I can see there are references to functions in other libraries. Can you remove this line from setup.py: extra_link_args=[] if DEBUG else ["-s", "-w"], Or otherwise make sure symbols are included If so, it'll probably give the specific function and line that fails

sergiomelero4OP•4w ago

Unfortunately I cannot debug or log while running on android 🙁 If I take that line away, it crashes automatically while running adb logcat. I've tried writting the logs on a file inside the phone, but it also crashes, I think it messes with kivy's clock or something.

Bækalfen•4w ago

I can't imagine how adding the debug symbols back in (removing the '-s -w' flags) would make it crash

sergiomelero4OP•2w ago

Hi guys! Just a little update, I'm still working on this: Now i can see which line of lcd.c is crashing.🥳 Which is: (unlikely(__pyx_get_slice_count(memview) <= 0)) As it only crashes in color mode and android, it is most likely caused by this memoryviews: self._tilecache1 = memoryview(self._tilecache1_raw).cast("B", shape=(TILES * 8, 8)) self._tilecache1_64 = memoryview(self._tilecache1_raw).cast("Q", shape=(TILES * 8,)) This kind of escapes my "technical level" 😆, but basically the b cast over a Q array only occurs once in the game boy color code in lcd.py, and is something that should not break in Desktop because it handles is automatically, but i can break in arm64 (Sorry for the poor explanation).

Bækalfen•7d ago

Great! Can you send your lcd.c? Also, do you have a line number?

sergiomelero4OP•7d ago

The crash line is 55185, thanks a lot for all the help! ❤️

lcd.c

Bækalfen•7d ago

Of course 😊 Do you have the rest of the stack trace? It’s a common function called from several places

sergiomelero4OP•7d ago

Yes, im extracting all the info! will be ready in a moment

sergiomelero4OP•7d ago

stacktrace_pyboy_and...

sergiomelero4OP•6d ago

tick() => scanline() => scanline_background() => tilecache = self._tilecache0 I believe all these problems happen because I'm using an old cython version (0.29.36, its the default one) in the buildozer environment. I tried the recommended one for pyboy once, but didnt make it work, maybe I'll try again now.

sergiomelero4OP•6d ago

So I put tilecache = _tilecache0[:] and its working for the first time (its still laggy, bc i got ride of a lot of optimizations so i could see logs) im gonna cry 🥺

Bækalfen•6d ago

Awesome! Very weird with that fix. I don't really know why PyBoy is very particular about the Cython version. But in that case, the compile should fail @sergiomelero4 What happens if you change this in lcd.pxd:

cdef int scanline_background(self, int, int, int, int, int, LCD) noexcept nogil
cdef int scanline_window(self, int, int, int, int, int, LCD) noexcept nogil

cdef int scanline_background(self, int, int, int, int, int, LCD) noexcept nogil
cdef int scanline_window(self, int, int, int, int, int, LCD) noexcept nogil

@cython.locals(tilecache=uint8_t[:,:])
cdef int scanline_background(self, int, int, int, int, int, LCD) noexcept nogil
@cython.locals(tilecache=uint8_t[:,:])
cdef int scanline_window(self, int, int, int, int, int, LCD) noexcept nogil

@cython.locals(tilecache=uint8_t[:,:])
cdef int scanline_background(self, int, int, int, int, int, LCD) noexcept nogil
@cython.locals(tilecache=uint8_t[:,:])
cdef int scanline_window(self, int, int, int, int, int, LCD) noexcept nogil

Or maybe add the 4 lines to CGBRenderer Interestingly, there is a difference between my lcd.c and yours. Yours call __PYX_XDEC_MEMVIEW(&__pyx_v_tilecache, 1); while mine calls __PYX_XCLEAR_MEMVIEW(&__pyx_v_tilecache, 0);. I think the crash happens because your code tries to decrement the reference count for an uninitialized memview. While mine correctly clears it. You also have a lot more checks and debug stuff. But that's maybe expected?

sergiomelero4OP•6d ago

This is the key to the thing! my cython version generates unsafer c code. PYX_XDEC_MEMVIEW is found in my cython release and PYX_XCLEAR_MEMVIEW is in the 3.0.6 one. I believe it is a known issue, this stackoverflow post mentions it: “The reference counting for memoryviews has been slightly rearranged for Cython 3.0 (mainly to avoid a few corner cases where references could be leaked) so it's not completely surprising that it's changed from 0.29.x.” https://stackoverflow.com/questions/76972950/cython-returned-memoryview-is-always-considered-uninitialized?utm_source=chatgpt.com

Stack Overflow

Cython returned memoryview is always considered uninitialized

Similar to Cython Memoryview as return value, but I didn't see a solution other than hacking the generated C code. I'm using Cython 3.0, but it looks like the result is the same with <3. Here's an

sergiomelero4OP•6d ago

unfourtunately when i tried to use the 3.0.6 numpy gave even more problems than pyboy 😆

sergiomelero4OP•6d ago

https://github.com/kivy/python-for-android/issues/2919 cython 3 upwards is not supported on python for android, guess I'll have to stick with 0.29.x for now

GitHub

Support Cython 3 · Issue #2919 · kivy/python-for-android

[This is an issue well-known to the developers, but I don't see it documented in the Issues database.] Cython 3 was released in July 2023. It isn't compatible with python-for-android and/or...

Bækalfen•3d ago

Interesting. At least you have a workaround for now I won't promise anything, but I'm trying to refactor the part you're having problems with @sergiomelero4 Check out the fix_n_perf branch. I've put my refactor there along with some unrelated stuff. Tell me how it goes https://github.com/Baekalfen/PyBoy/tree/fix_n_perf

sergiomelero4OP•3d ago

Nice!!! I'll try it after work!

sergiomelero4OP•2d ago

I'm merging it with the changes which make it compatible with old cython/android (everything but the [:] fix) https://github.com/sergiomele97/PruebaPyBoyNperf/tree/fix_n_perf

GitHub

GitHub - sergiomele97/PruebaPyBoyNperf at fix_n_perf

Game Boy emulator written in Python. Contribute to sergiomele97/PruebaPyBoyNperf development by creating an account on GitHub.

sergiomelero4OP•2d ago

ouch! It started the game correctly but i get a consistent crash, before the one I used to have:

message.txt

Bækalfen•2d ago

What's the purpose of your "onlyRAM" changes? I can't really see from the trace where the code failed Is it really needed to remove nogil?

sergiomelero4OP•2d ago

So android has grown a little "special" about permissions and the default RAM save on pyboy.close() doesnt work, so i added a parameter on save_state so we can replicate that behaviour (saving just the ram) whenever we want, also not needing to end the emulation each time. I think we could even merge it on the master branch.

Bækalfen•2d ago

I see. How about you save the full state when the emulator is running -- app is minimized, phone sleeps etc. And when you're signalled to fully close, you call pyboy.stop(save=True, path=...) and I add a path or file-like object (io.BytesIO()) argument?

sergiomelero4OP•2d ago

Yes, sorry! I have to add again some changes so we get a better stack trace, I'll do it as soon as posible! I'm not 100% sure, but i believe its because of the old cythom version :C

Bækalfen•2d ago

I guess we could have an optional pyboy.save(...) maybe Hmm. What error do you see? nogil is an ancient feature

sergiomelero4OP•2d ago

It tells me in a lot of places that the operation is not allowed without gil

Bækalfen•2d ago

Can you paste one of them? Does it have to do with logging? Because I have a hack in setup.py that takes care of that

sergiomelero4OP•this hour

Sorry i dont have access to the computer right now, i'll paste it latter 😅 I'm not sure it has to do with logging Yes!! That's where i currently am 😁 I believe the path argument would work aswell

sergiomelero4OP•20h ago

Here's the stacktrace as promised 😁

stacktrace2.txt

sergiomelero4OP•20h ago

And here are the logs I get if I dont remove the nogil

compiling_error.txt

Bækalfen•18h ago

There's something wrong with your cython c-files. It's not producing the correct C-types. Maybe that's why you need to change so many things? Try making a syntax error in the lcd.pxd file? Maybe they are not picked up. Like this is what mine produces:

    if (__pyx_v_self->cgb) {
      __pyx_t_3 = __pyx_v_sprite_cache_no;
    } else {
      __pyx_t_3 = 0;
    }
    ((struct __pyx_vtabstruct_5pyboy_4core_3lcd_Renderer *)__pyx_v_self->__pyx_vtab)->update_spritecache(__pyx_v_self, __pyx_v_sprite_cache_no, __pyx_v_lcd, __pyx_v_tileindex, __pyx_t_3);

    if (__pyx_v_self->cgb) {
      __pyx_t_3 = __pyx_v_sprite_cache_no;
    } else {
      __pyx_t_3 = 0;
    }
    ((struct __pyx_vtabstruct_5pyboy_4core_3lcd_Renderer *)__pyx_v_self->__pyx_vtab)->update_spritecache(__pyx_v_self, __pyx_v_sprite_cache_no, __pyx_v_lcd, __pyx_v_tileindex, __pyx_t_3);

Notice the lack of PyObject and ref-counting. Now, this is yours:

    if ((__pyx_v_self->cgb != 0)) {
      __pyx_t_13 = ((PyObject *)__pyx_v_sprite_cache_no);
    } else {
      __pyx_t_13 = ((PyObject *)0);
    }
    ((struct __pyx_vtabstruct_5pyboy_4core_3lcd_Renderer *)__pyx_v_self->__pyx_vtab)->update_spritecache(__pyx_v_self, __pyx_v_sprite_cache_no, __pyx_v_lcd, __pyx_v_tileindex, ((int)__pyx_t_13));
    __Pyx_DECREF(__pyx_t_13); __pyx_t_13 = 0;

    if ((__pyx_v_self->cgb != 0)) {
      __pyx_t_13 = ((PyObject *)__pyx_v_sprite_cache_no);
    } else {
      __pyx_t_13 = ((PyObject *)0);
    }
    ((struct __pyx_vtabstruct_5pyboy_4core_3lcd_Renderer *)__pyx_v_self->__pyx_vtab)->update_spritecache(__pyx_v_self, __pyx_v_sprite_cache_no, __pyx_v_lcd, __pyx_v_tileindex, ((int)__pyx_t_13));
    __Pyx_DECREF(__pyx_t_13); __pyx_t_13 = 0;

The compile error file is empty

Gaming

Programming

Brainstorming: adapting lcd.py to android

Did you find this page helpful?