devkitpro / newlib Goto Github PK

Hi, many homebrews need you to add support for https in order to be able to donwload files from servers like google drive that only support https protocol. Please could you add curl with https support

Hello! I noticed with the latest devkitARM update (r60-9) that newlib was updated to 4.3.0 and there appears to be a full pthread API included as part of devkitPro/buildscripts@cd5e224

As far as I can tell, the syscalls used by the pthread implementation are not actually implemented anywhere, so none of the pthread_* functions actually work, which is fine - however, it does seem like on the 3DS it would be possible to implement some of the functionality using libctru calls and get a mostly-working pthread implementation.

In light of this, I had a few questions about the new changes and any future plans for them:

1. Is there an intention to support pthreads on the 3DS?
   • If so, would the syscalls be implemented in libctru or perhaps somewhere else? Are the maintainers looking for help implementing this?
   • If not, is this a supported scenario (for users that are willing to write syscalls themselves)?
2. Will the patch linked above eventually be delivered to the devkitARM branch of this repo? If not, is there somewhere else it lives that can be used for reference, besides just the buildscripts repo as a big diff?

Thanks in advance! I was excited to see these changes pulled down when I upgraded, but I wanted to set my expectations and see what, if any plans you all had for these APIs.

The file newlib/libc/include/termios.h of the devkitA64 branch (maybe others?) includes <sys/termios.h> which is not included or defined anywhere, which causes code that includes termios.h to fail compilation.

I was in the process of porting OpenSSL to devkitA64 in which I found that the path of least resistance to getting the port compiling was to simply create a blank sys/termios.h file. What do you think is the best approach for handling this termios.h file, since in its current state can only cause compilation errors when included? Short of fully implementing sys/termios.h, should a blank sys/termios.h file be created; termios.h be changed to do nothing; remove termios.h altogether?

As analyzed in this other conversation, in the commit 3c75fac (done by upstream) the logic whether to define or not __restrict keyword was changed, and now it is assumed to always be existing even in c++ mode, thus it's not redefined as an empty macro.
Unfortunately the gcc shipped with dkp doesn't seem to support that keyword at all, even when compiling with gnu extensions (with -std=gnu++XX).

#include <sys/types.h>
#include <regex.h>

int main() {
    return 0;
}

Attempting to compile this sample code as cpp will error out

In file included from C:/devkitPro/examples/nds/templates/arm9/source/main.cpp:2:
C:/devkitPro/devkitARM/arm-none-eabi/include/regex.h:99:45: error: expected primary-expression before '__restrict'
   99 |                         size_t, regmatch_t [__restrict], int);
      |                                             ^~~~~~~~~~
C:/devkitPro/devkitARM/arm-none-eabi/include/regex.h:99:45: error: expected ']' before '__restrict'
   99 |                         size_t, regmatch_t [__restrict], int);
      |                                             ^~~~~~~~~~
      |                                             ]
C:/devkitPro/devkitARM/arm-none-eabi/include/regex.h:99:45: error: expected ')' before '__restrict'
   99 |                         size_t, regmatch_t [__restrict], int);
      |                                             ^~~~~~~~~~
      |                                             )
C:/devkitPro/devkitARM/arm-none-eabi/include/regex.h:98:16: note: to match this '('
   98 | int     regexec(const regex_t *__restrict, const char *__restrict,
      |                ^
C:/devkitPro/devkitARM/arm-none-eabi/include/regex.h:99:55: error: expected initializer before ']' token
   99 |                         size_t, regmatch_t [__restrict], int);

In this case the __restrict keyword is being used by the regex.h header, another component of newlib.

aligned_alloc is defined to use posix_memalign:
https://github.com/devkitPro/newlib/blob/devkitPPC_r31/newlib/libc/stdlib/aligned_alloc.c

aligned_alloc is a stanadard C11 function:
http://en.cppreference.com/w/c/memory/aligned_alloc

I can confirm memalign does work so I guess maybe the fix would be to change aligned_alloc to use memalign instead of posix_memalign? I'm not sure so that's why I went for an Issue instead of a Pull request.

In devkitPPC release 29, the compiler gives a warning for "ftruncate" (undefined function).
However the compiler can link the function.

This is due to sys/unistd.h
#if defined(CYGWIN) || defined(rtems) || defined(aarch64) || defined (arm) || defined(sh) || defined(SPU)
#if !defined(INSIDE_CYGWIN)
int _EXFUN(ftruncate, (int __fd, off_t __length));
int _EXFUN(truncate, (const char *, off_t __length));
#endif
#endif

where PPC architecture is not included in the "#if defined" list.

Hello,
(This issue might be out of newlib’s scope)
I m currently facing an issue with the pthread_create interface.

Despite mutex/barrier and sched_yield some threads never execute after reaching a specific number. Some threads are like frozen (hanged).

The number of running concurrent threads look limited.
Is there a way to increase this limit ?

I created the following snippet to isolate my issue.
https://gist.github.com/H0neyBadger/c080dd00c0dd77439bd8d41826d49d0c#file-main-cpp-L131
With thread count to 68, the job is running as expected.
With thread count to 69 (or greater) one or many threads are frozen. The thread barrier is never reached.

I m using nxlink and homebrew netloader to test my projects (with the latest devkitpro pacman on arch).

I spent a lot of time on this issue, and I m out of idea.
I hope you can help

I consistently got an error "Building with plugin support requires a host that supports dlopen" when compiling binutils 2.27 from your distribution. I was able to get everything to work by updating to 2.27.90 from upstream (I still have it apply your opcodes patch).

The good news is I appear to have a working 64 bit toolchain on Sierra.

Thanks for this excellent project!

Hi there!
I hope that this is the right place for reporting this: I see that devkitPro (and more precisely, the dkp-linux package) installs a file

/opt/devkitpro/devkitPPC/powerpc-eabi/include/machine/endian.h

which I hoped would also contain the same function that are listed here: https://man7.org/linux/man-pages/man3/endian.3.html and which can be used to convert to/from big/little endian. I know that SDL also offers the same functionality, but since I need them in some code which is not SDL-specific, I'd rather use endian.h.

I wonder, would it be possible to add those functions to the header file? I know that they are non standard, but they are also quite useful :-)

With the latest version of devkitpro, compiling and running the following code results in an error. However, when using an older version (newlib 4.2.0), it runs fine.

This code continuously calls vsnprnitf from both the main and another thread:

#include <cstdarg>
#include <cstdio>
#include <thread>

#include <switch.h>

static bool running = true;

void print(const char *format, ...) {
    char data;
    va_list args;
    va_start(args, format);
    vsnprintf(&data, 1, format, args);
    va_end(args);
}

static void startPrint() {
    while (running) {
        print("%f\n", 1.0);
    }
}

int main(int argc, char **argv)
{
    consoleInit(NULL);
    padConfigureInput(1, HidNpadStyleSet_NpadStandard);
    PadState pad;
    padInitializeDefault(&pad);
    printf("\x1b[16;20HHello World!");

    // start a new thread to call `print`
    std::thread printThread(startPrint);
    
    while(appletMainLoop())
    {
        padUpdate(&pad);
        u64 kDown = padGetButtonsDown(&pad);
        if (kDown & HidNpadButton_Plus) break;
        consoleUpdate(NULL);

        // call `print` at main thread
        print("%f\n", 1.0);
    }

    running = false;
    if (printThread.joinable()) printThread.join();
    consoleExit(NULL);
    return 0;
}

Here is the project code that includes the makefile:
hello-world.zip

I am trying to find out exactly where the problem is, but unfortunately, due to my limited understanding of newlib, there has been no progress so far.

Today I migrated my environment to pacman based one, using devkitARM r48-2 and I found some of my code failed to build. I found the return type of ftell in the file arm-none-eabi/include/stdio.h is off_t rather than long. This is not conforming to any version of ISO C and POSIX. The document of newlib I've found also shows it should be long.

Strangely, the code in

What would be involved in getting the C++ standard library to work with C++11's std::thread, std::mutex and std::condition_variable? I would like to write some home brew that has a code base that uses these C++ features.

I have at least two libraries I'm trying to build that are trying to use fnmatch. One is configure based and the other is CMake based. The CMake based one is assuming the feature is there and not checking it (which sucks). For both it finds fnmatch.h but gets a linker error later. I can build it into my library (which is a bad idea) or hack the configure script output to not define HAVE_FNMATCH_H like so:

sed -i -e 's/#define HAVE_FNMATCH_H 1//' aarch64-none-elf/config.h

That is not ideal though. Is there a downside to building the posix implementation into libc? I see that opendir is still in libsysbase (hence the HAVE_OPENDIR define) after I make this change and half the posix files can be disabled or don't build anyway.

diff --git a/newlib/configure.host b/newlib/configure.host
index d536e0d..5c971a1 100644
--- a/newlib/configure.host
+++ b/newlib/configure.host
@@ -409,6 +409,9 @@ case "${host}" in
        sys_dir=a29khif
        signal_dir=
        ;;
+  aarch64*-*-*)
+    posix_dir=posix
+    ;;
   arm*-*-*)
        sys_dir=arm
        if [ "x${newlib_may_supply_syscalls}" = "xno" ] ; then
@@ -627,7 +630,7 @@ newlib_cflags="${newlib_cflags} -DCLOCK_PROVIDED -DMALLOC_PROVIDED -DEXIT_PROVID
        default_newlib_io_c99_formats="yes"
        default_newlib_io_long_long="yes"
        default_newlib_io_pos_args="yes"
-       newlib_cflags="${newlib_cflags} -D__DYNAMIC_REENT__ -DREENTRANT_SYSCALLS_PROVIDED -D__DEFAULT_UTF8__"
+       newlib_cflags="${newlib_cflags} -DHAVE_OPENDIR -D__DYNAMIC_REENT__ -DREENTRANT_SYSCALLS_PROVIDED -D__DEFAULT_UTF8__"
        newlib_cflags="${newlib_cflags} -ffunction-sections -fdata-sections"
        syscall_dir=syscalls
        ;;

mkdir used as an example as it's where the bug was first noticed.

https://github.com/devkitPro/newlib/blob/devkitA64/libgloss/libsysbase/iosupport.c#L70
https://github.com/devkitPro/newlib/blob/devkitA64/libgloss/libsysbase/iosupport.c#L80
When provided with the path, these lines then assume that the first colon it finds is separating the device from the path, which isn't necessarily true as relative paths don't include any devices, so the function ends up trying to resolve the first half before the colon as a device resulting in it returning -1 making mkdir read at index -1 of devoptab_list causing a segfault, this likely affects other fs functions.

This was already discussed with @Stary2001 on Discord. The issue here is just for the record in case someone else struggles with this in the meantime.

The current implementation of scandir in devkitA64 fails if errno is non-zero at the time when scandir is called.

Expected behavior: the result of scandir should not depend on any prior value of errno.

Proposed fix: set errno to zero at the beginning of scandir.

Workaround for now: replace scandir(...) with errno = 0; scandir(...)

While investigating this downstream bug, I've noticed an issue with a patch devkitPro does to newlib's fread function: https://bugs.scummvm.org/ticket/11342

The program does the following calls to trigger data corruption:

	char *buffer = (char *)malloc(20*1024);

	// Open a file for reading with buffered IO
	FILE *f = fopen("sdmc:/ScummVM/The Dig/DIG.LA1", "r");

	// Go somewhere in the file, unimportant
	fseek(f, 23800663, SEEK_SET);
	// Initialize the stdio read buffer
	fread(buffer, 1, 1, f);
	// Make a large read that enters devkitPro's fread patched code path.
	// After this, the stdio read buffer is out of sync with the stream position.
	fread(buffer, 1, 10096, f);
	// Seek back a bit in the file. The target seek position is inside the buffer from
	// stdio's perspective (but actually is not as the buffer is out of sync with the actual
	// stream position).
	fseek(f, 23810752, SEEK_SET);
	// Make a small read. The data is fetched from the stdio buffer. It is inconsistent with what is actually in the file.
	fread(buffer, 1, 4, f);

	fclose(f);

I suggest reverting this patch:

As you can see at https://github.com/devkitPro/newlib/blob/devkitPPC/libgloss/libsysbase/dirent.c#L172, it will return DT_REG if a devoptab returns DT_LNK or DT_UNKNOWN.

I've found this problem while working on improved devoptabs for wut and libiosuhax for the Wii U to support the Wii U's symlink variant. It'd help a lot since currently even though it doesn't return DT_UNKNOWN you have to ignore it and call lstat to check for the proper type.

I can make a pull request, but I wasn't too sure which branch I should apply it too so I thought I should open an issue first. I've tested it and if we used the same macro from https://github.com/devkitPro/newlib/blob/devkitPPC/winsup/cygwin/include/sys/dirent.h#L89-L90 it would work for all types properly.

<sys/un.h> header is missing.
https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/un.h.html

It is needed for the asio library.

earlier toolchains defaulted to C.UTF-8, r29 does not

Error compiling libpng-1.6.28 (portlibs)

Makefile generated with this script:

---cut here
#!/bin/bash

PATH=${DEVKITPPC}/bin:${PATH}
HOST=powerpc-eabi
PREFIX=${DEVKITPRO}/portlibs/ppc

#libpng
cd libpng*
CPPFLAGS=-I${PREFIX}/include LDFLAGS=-L${PREFIX}/lib ./configure
--prefix=${PREFIX} --host=${HOST} --disable-shared --enable-static
make && make install
cd ${home}

---cut here

This is the output:

$ make
make all-am
make[1]: Entering directory /c/Users/Nebiun/Downloads/WIIdev/portlibs_2017/libp ng-1.6.28' /bin/sh ./libtool --tag=CC --mode=link powerpc-eabi-gcc -g -O2 -L/c/devkitP ro/portlibs/ppc/lib -o pngcp contrib/tools/pngcp.o libpng16.la -lz -lm libtool: link: powerpc-eabi-gcc -g -O2 -o pngcp contrib/tools/pngcp.o -L/c/devk itPro/portlibs/ppc/lib ./.libs/libpng16.a -lz -lm c:/devkitpro/devkitppc/bin/../lib/gcc/powerpc-eabi/6.3.0/../../../../powerpc-eab i/bin/ld.exe: warning: cannot find entry symbol _start; defaulting to 018000b8 contrib/tools/pngcp.o: In function isdir':
c:\Users\Nebiun\Downloads\WIIdev\portlibs_2017\libpng-1.6.28/contrib/tools/pngcp
.c:1610: undefined reference to access' contrib/tools/pngcp.o: In function cp_one_file':
c:\Users\Nebiun\Downloads\WIIdev\portlibs_2017\libpng-1.6.28/contrib/tools/pngcp
.c:2176: undefined reference to access' c:\Users\Nebiun\Downloads\WIIdev\portlibs_2017\libpng-1.6.28/contrib/tools/pngcp .c:2207: undefined reference to access'
collect2.exe: error: ld returned 1 exit status
make[1]: *** [pngcp] Error 1
make[1]: Leaving directory `/c/Users/Nebiun/Downloads/WIIdev/portlibs_2017/libpn
g-1.6.28'
make: *** [all] Error 2

I'm using the latest dpkg and cmake

devilutionX/3rdParty/StormLib/src/StormPort.h:279:23: error: 'ftruncate' was not declared in this scope

<unistd.h> is included

There isn't any ability to get ctime or atime from the 3DS and mtime can only be retrieved by using a path so it can only be filled in for stat and not fstat. There's also resistance to adding it due to it being slow. See devkitPro/libctru#458 (comment)

I'm requesting that the header file be updated to flag the compiler to print out a warning if someone is using st_mtime, st_atime, or st_ctime in their code so that they can take appropriate action. Alternatively they could just not be defined for the 3DS but then that breaks POSIX requirements.

One possibility is adding in the deprecated pragma and then a comment in the header file along with what to do for the 3DS (i.e. call archive_getmtime instead): https://stackoverflow.com/questions/22823477/c-portable-way-to-deprecate-struct-members

<sys/uio.h> header is missing.
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_uio.h.html

It is needed for the asio library.