Copied from SourceForge bug report #200 (Reported my me also)

Building flex without libfl is useful when you know that this flex build of yours will be used only with a cross-toolchain. For cross-building environments, having a libfl built for the host architecture is not useful. (We need one for the target architecture instead.)

For my case, although I would need to configure and build flex twice for a cross environment, I need to use a hack to disable libfl currently:

sed -i.bak -e 's/lib_LTLIBRARIES = \$/lib_LTLIBRARIES =/' \
    -e 's/\tlibfl.la \$/#&/' -e 's/\tlibfl_pic.la$/#&/' src/Makefile.in

It's better to make it into a configure option. (For your reference, GNU Bison can disable building liby by the '--disable-yacc' option.)

Alternatively, add a configure-time variable to explicitly specify which compiler will be used to build libfl (could be named CC_FOR_TARGET or CC_FOR_LIBFL). If using this approach, don't forget the respective CFLAGS_FOR_TARGET and other variables.

Traditional C support involves a lot of cruft in flex.skl. Flex itself uses function prototypes, so one must have at least a C89 compiler just to build Flex.

M4 is standard on Unix, but not on Windows, and may not be easy to compile on Windows.

It would be nice to not need it.

The SourceForge site no longer has it, and it's not in any obvious place on the GitHub site.

Hello,

The new version of flex 2.6.2 requires help2man to generate the man page for flex. This will be fail if you are doing a GNU/Linux from scratch, which is my case, because flex needs to be available early, and help2man depends on perl, and perl is available later. (See also "Linux From Scratch").

P.S: Thanks for provide flex in lzip format.

flex 2.6.1-1 on Arch Linux generates C that produces compiler warnings
with gcc.

In function ‘yy_get_next_buffer’:
error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
   for ( n = 0; n < max_size && \
                  ^
note: in expansion of macro ‘YY_INPUT’
   YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),

In function ‘yy_scan_bytes’:
error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
  for ( i = 0; i < _yybytes_len; ++i )
                 ^

Hi,

This issue was reported by a Debian user in Bug 729927

The command
flex --noline name.lex

generates the line
#line 3 "lex.yy.c"

This causes some confusion when:
1: line 3 is really line 2, and,
2: when (in order to compile using C++) the file has
been renamed to (something like) lex.yy.cpp causing
error messages to specify file lex.yy.c instead of
lex.yy.cpp (or some other name).

Manoj

Since 84ca965, if help2man is not installed configure will warn but make install will fail since flex.1 is never actually generated.

Ideally either the build would succeed without installing the man page or configure would fail if help2man was not installed.

EDIT: I seem to be completely wrong here. I'll take another look at what was causing the failure.

Testing 2.6.1 on cygwin to allow the tests to be executed I need to
create links like: alloc_extra.exe.txt -> alloc_extra.txt

After the workaround
for i in *.txt ; do ln -s $i ${i/txt/exe.txt} ; done
the test can be run
Testsuite summary for the fast lexical analyser generator 2.6.1

TOTAL: 114
PASS: 68
SKIP: 0
XFAIL: 0
FAIL: 46
XPASS: 0
ERROR: 0

Hi,

This bug was reported by a Debian user at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616099

If you ask for a reentrant scanner with yylineno, nothing initialises yylineno for you. This is not clearly documented AFAICT.

So you have to explictly call yyset_lineno. I think it would be convenient if yylex_init did this for you. Ie, yy_init_globals should clear yylineno.

Manoj
libxlu_cfg_l.l.txt

libfl_pic.a contains the same non-PIC object files as libfl.a, not the PIC object files (tested building flex 2.6.1 on Debian).

If this didn't cause build breakage anywhere, this kinda implies that noone actually needs libfl_pic.a

Since 2.6.0, flex now needs bison during build.
Before that, at least on NetBSD, yacc was enough.

The error I see is

gmake[1]: Entering directory '/scratch/devel/flex/work/flex-2.6.0/tests'
/bin/sh ../build-aux/ylwrap bison_nr_parser.y y.tab.c bison_nr_parser.c y.tab.h `echo bison_nr_parser.c | sed -e s/cc$/hh/ -e s/cpp$/hpp/ -e s/cxx$/hxx/ -e s/c++$/h++/ -e s/c$/h/` y.output bison_nr_parser.output -- yacc -d -p test
yacc: e - line 81 of "/scratch/devel/flex/work/flex-2.6.0/tests/bison_nr_parser.y", syntax error
        if( $1 != @1.first_line || $1 != testget_lineno())
                  ^
Makefile:2075: recipe for target 'bison_nr_parser.c' failed
gmake[1]: *** [bison_nr_parser.c] Error 1
../src/flex -o bison_nr_scanner.c bison_nr_scanner.l
/bin/sh ../build-aux/ylwrap bison_yylloc_parser.y y.tab.c bison_yylloc_parser.c y.tab.h `echo bison_yylloc_parser.c | sed -e s/cc$/hh/ -e s/cpp$/hpp/ -e s/cxx$/hxx/ -e s/c++$/h++/ -e s/c$/h/` y.output bison_yylloc_parser.output -- yacc -d -p test
yacc: e - line 83 of "/scratch/devel/flex/work/flex-2.6.0/tests/bison_yylloc_parser.y", syntax error
        if( $1 != @1.first_line || $1 != testget_lineno(scanner))
                  ^
Makefile:2075: recipe for target 'bison_yylloc_parser.c' failed
gmake[1]: *** [bison_yylloc_parser.c] Error 1
../src/flex -o bison_yylloc_scanner.c bison_yylloc_scanner.l
/bin/sh ../build-aux/ylwrap bison_yylval_parser.y y.tab.c bison_yylval_parser.c y.tab.h `echo bison_yylval_parser.c | sed -e s/cc$/hh/ -e s/cpp$/hpp/ -e s/cxx$/hxx/ -e s/c++$/h++/ -e s/c$/h/` y.output bison_yylval_parser.output -- yacc -d -p test
updating bison_yylval_parser.h

This is a problem because bison currently depends on flex, so we have
a dependency cycle.

As a workaround, I've locally disabled the bison tests:

--- tests/Makefile.in.orig  2015-11-17 16:19:46.000000000 +0000
+++ tests/Makefile.in
@@ -131,8 +131,8 @@ CONFIG_HEADER = $(top_builddir)/src/conf
 CONFIG_CLEAN_FILES =
 CONFIG_CLEAN_VPATH_FILES =
 am__EXEEXT_1 = alloc_extra$(EXEEXT) array_nr$(EXEEXT) array_r$(EXEEXT) \
-   basic_nr$(EXEEXT) basic_r$(EXEEXT) bison_nr$(EXEEXT) \
-   bison_yylloc$(EXEEXT) bison_yylval$(EXEEXT) c_cxx_nr$(EXEEXT) \
+   basic_nr$(EXEEXT) basic_r$(EXEEXT) \
+   c_cxx_nr$(EXEEXT) \
    c_cxx_r$(EXEEXT) ccl$(EXEEXT) cxx_basic$(EXEEXT) \
    cxx_multiple_scanners$(EXEEXT) debug_nr$(EXEEXT) \
    debug_r$(EXEEXT) extended$(EXEEXT) header_nr$(EXEEXT) \
@@ -1440,12 +1440,6 @@ string_r_SOURCES = string_r.l
 top_SOURCES = top.l top_main.c
 yyextra_SOURCES = yyextra.l
 BUILT_SOURCES = \
-   bison_nr_parser.h \
-   bison_nr_scanner.h \
-   bison_yylloc_parser.h \
-   bison_yylloc_scanner.h \
-   bison_yylval_parser.h \
-   bison_yylval_scanner.h \
    header_nr_scanner.h \
    header_r_scanner.h \
    multiple_scanners_nr_1.h \

Perhaps the configure script can test for bison availability and only
enable the tests in case bison is indeed available.

For a reentrant lexer YY_CURRENT_BUFFER_LVALUE contains as members the current line number, but also the current column number. It would be great if the macro YY_LESS_LINENO could take care of yycolumn, too.
For now I am currently using the snippit below in the .l footer as a workaround which might be a starting point for a patch supposedly to flex.skl:

--- YY_LESS_LINENO_v0.c
+++ YY_LESS_LINENO_v1.c
 #undef YY_LESS_LINENO
 #define  YY_LESS_LINENO(n) \
             do { \
                 int yyl;\
-                for ( yyl = n; yyl < yyleng; ++yyl )\
-                    if ( yytext[yyl] == '\n' )\
-                        --yylineno;\
+                for ( yyl = n; yyl < yyleng; ++yyl ){\
+                    yycolumn--; \
+                    if ( yytext[yyl] == '\n' ){\
+                        --yylineno;yycolumn=0;}\
+                }\
             }while(0)

Thanks,
J.

The following simple lexer produces the wrong output:

%{
#include <stdio.h>
%}
%%
(.|\n) { puts("YY_G(alpha)"); }
%%

When run, it produces (alpha) for every input character, instead of YY_G(alpha).

This is due to M4 macro expansion and is a bug. Namespaces are not an issue here, since YY_G occurs in a string literal.

This can be fixed by quoting all user-supplied input.

Hi,

This feature request was submitted by a Debian user in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=603787

since/when the running lexer (not only reentrant) does record the location, please make yy_fatal_error print it by default.

Thank you,

Manoj

...so if you go to the main GitHub page, it points you to the SourceForge site if you scroll down.

On this line, the ^ should be turned into ^, so it's negating the character class.

   "{"[\^{}}\n]*"}"     /* eat up one-line comments */

I ran this on my machine, and parsing Pascal-style comments like:

{ this is a comment }

doesn't work without the change. It treats the comments as code.

I didn't see the flex-www repo migrated to Github. If I missed it, please point me there and I can send you a pull request if desired.

http://flex.sourceforge.net/manual/Simple-Examples.html#Simple-Examples

http://flex.cvs.sourceforge.net/viewvc/flex/flex-www/manual/Simple-Examples.html?revision=1.3&view=markup

Hello (again),

The following piece of code (see below) is ok under
flex (2.6.1 or 2.6.2) but still produces annoying warnings
when given to the -Wswitch-default option of gcc (i'm using gcc-6.2.0):

cm2.cc: In function 'int yylex()':
cm2.cc:814:15: warning: switch missing default case [-Wswitch-default]

cm2.cc: In function 'int yyinput()':
cm2.cc:1140:11: warning: switch missing default case [-Wswitch-default]

Addition of a simple 'default: break;' at various places inside the
flex code (2.6.1 or 2.6.2) corrects the problem.

%{
%}
%S INSIDE
%%
"beg of inside" { ECHO; BEGIN INSIDE; };
"end of inside" { ECHO; BEGIN INITIAL; };
%%

Just as bison has a yacc-compatibility mode, flex should have a way to build lex-compatible files, while also allowing for modern, correct code for cases where the old lex way is incorrect or broken.

Hi,

This bug was reported by a Debian user in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642040

I'm trying to compile libsbook from here: http://simson.net/ref/sbook5/
During compilation, i get:

 flex -tL8B  -i parse_address.flex  | sed s/yy/yyaddress/g | grep -v unistd.h > parse_address.cpp
 flex: input rules are too complicated (>= 32000 NFA states)

A web search points me to this FAQ:
http://flex.sourceforge.net/manual/unnamed_002dfaq_002d88.html#unnamed_002dfaq_002d88
which contains this suggestion from Vern Paxson:

Increase the definitions in flexdef.h for:

define JAMSTATE -32766 /* marks a reference to the state that always jams */

define MAXIMUM_MNS 31999

define BAD_SUBSCRIPT -32767

recompile everything, and it should all work.

Is there a reason to avoid making these changes in flex? Or can this be chosen as a run-time option? Processing power even on netbooks is demonstrated to be enough to handle instructions much more complex than this.

Manoj

Hello,

The following piece of code (see below) was ok
under flex-2.6.1, but now outputs:

cm1.lex:11: EOF encountered inside an action

under flex-2.6.2. Please note the quote within
the //-kind-of comment

%{
%}
%S INSIDE
%%
"" {
// '
ECHO; BEGIN INSIDE;
};
"" { ECHO; BEGIN INITIAL; };
%%

To someone not familiar with flex it looks like the homepage is still http://flex.sourceforge.net/ (this is what google and essentially all the links on the web say) and that page implies that downloads happen from sourceforge --and the download page says that the newest version is 2.6.0.

This is downright dangerous (as it leads people to believe there are no security updates): the page should be updated to link to github.

As of 53e3088, flex runs stage1flex to bootstrap itself. Unfortunately, this fails when cross compiling, as stage1flex is built for the target architecture (not for the host machine).

I'm unsure how to fix this as my knowledge of GNU autotools is limited; but http://lists.gnu.org/archive/html/automake/2008-12/msg00016.html seems to suggest a way to make this work.

Add an option, say, --with-m4 to configure that allows a user to specify which m4 flex should use. This option, when passed, should cause a symbol to be set in config.h. src/main.c should use that symbol to call m4. If that option is not passed, then configure should do its best to find a suitable m4 and set the config.h symbol as previously described.

In particular, this effects Solaris, where /usr/bin/m4 is not suitable for use with flex and where GNU m4 is not in a standard place.

Have some very old releases been deleted recently? My project still relies on 2.5.4a. Is there any chance they could be restored if the effort involved is small? I think it would be good to have them somewhere on the net for historical purposes anyway.

Backtrace and log file (github didn't let me attach it) follow below:

(gdb) bt
#0  testFlexLexer::yy_init_buffer (this=this@entry=0x7fffc91999a0, b=0xca5c20, file=...) at cxx_restart.cc:1319
#1  0x00000000004029e1 in testFlexLexer::yyrestart (this=this@entry=0x7fffc91999a0, input_file=...) at cxx_restart.cc:1191
#2  0x00000000004011b8 in testFlexLexer::yyrestart (input_file=0x0, this=0x7fffc91999a0) at cxx_restart.cc:1202
#3  main () at cxx_restart.ll:48

#!/bin/bash -vx
set -euo pipefail
+ set -euo pipefail

# testwrapper.sh: run a flex test, typically called by a Makefile

# Each test will exercise some feature or aspect of flex. Run the test with any input it may need.

INPUT_DIRECTORY=""
+ INPUT_DIRECTORY=
INPUT_NAME=""
+ INPUT_NAME=
INPUT_COUNT=0
+ INPUT_COUNT=0
USE_REDIRECT=0
+ USE_REDIRECT=0
DO_COMPARISON=0
+ DO_COMPARISON=0

while getopts :d:i:rt1 OPTION ; do
    case $OPTION in
        d) INPUT_DIRECTORY=$OPTARG ;;
        i)
            if [ "$INPUT_NAME" == "" ] ; then
                INPUT_NAME="$OPTARG"
            else
                INPUT_NAME="$INPUT_NAME $OPTARG"
            fi
            INPUT_COUNT=$(($INPUT_COUNT+1))
            ;;
        r) USE_REDIRECT=1 ;;
        t) USE_TABLES=1 ;;
        1) DO_COMPARISON=1 ;;
    esac
    done
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ INPUT_DIRECTORY=.
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_REDIRECT=1
+ getopts :d:i:rt1 OPTION

TESTNAME="${!OPTIND}"
+ TESTNAME=./cxx_restart

INPUT_NAME=${INPUT_NAME:-$INPUT_DIRECTORY/`basename ${TESTNAME%.exe}`.txt}
++ basename ./cxx_restart
+ INPUT_NAME=./cxx_restart.txt

if [ "$DO_COMPARISON" -eq "1" ] ; then
    test `$TESTNAME 1 < $INPUT_NAME` -eq `$TESTNAME < $INPUT_NAME`
    exit $?
    fi
+ '[' 0 -eq 1 ']'

if [ $INPUT_COUNT -gt 1 ] ; then
    $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} ${INPUT_NAME}
    exit $?
    fi
+ '[' 0 -gt 1 ']'

if [ -f ${INPUT_NAME} ] ; then
    if [ $USE_REDIRECT == 1 ] ; then
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
    else
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} $INPUT_NAME
    fi
else
    $TESTNAME
fi
+ '[' -f ./cxx_restart.txt ']'
+ '[' 1 == 1 ']'
+ ./cxx_restart
./testwrapper.sh: line 53: 22774 Speicherzugriffsfehler  (core dumped) $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
FAIL cxx_restart (exit status: 139)

yyleng is treated throughout as a "size_t"-equivalent type; it is not correct to type it as an "int". This change was made sometime between 2.6.0 and 2.6.2, and it breaks existing code.

This should be much easier if Flex is ported to C++ (see #77).

With version 2.6.1 if prefix was set, then all affected functions would have their declaration changed and all references in the lexer input file would be changed as well. With v2.6.2 only the function declarations are changed.

For example, set prefix to AA, reference yyget_extra() in the lexer input file, and build the C file using v2.6.1 and all calls in the C source file will be to AAget_extra(). However, build with v2.6.2 and the resultant C file will still have calls to yyget_extra() resulting in a broken source file.

This change in behavior appears to have been introduced with commit ee87e5d.

Hello,

Please consider cm41.lex and cm42.lex (see below). They are the
same, except for an additional line (namely a comment in this case) after
the first %%.

Run flex, compile and run (under some recent flex version).

./cm41 produces:
cm41.lex 10
while ./cm42 produces:
cm42.lex 12

Conclusion:

• cm41 is correct since the printf() is indeed located line 10 of cm41.lex
• cm42 is wrong since the printf() is indeed located line 11 of cm42.lex

Adding more (indented) lines immediately after the first %% does not
increase the mismatch. See also Debian bug #542482.

This is cm41.lex:
%{
%}
%S INSIDE
%%
"<inside>" { ECHO; BEGIN INSIDE; };
<INSIDE>"</inside>" { ECHO; BEGIN INITIAL; };
%%
int yywrap() { return(1); };
int main() {
printf("%s %d\n", FILE, LINE);
//yylex();
};

This is cm42.lex:
%{
%}
%S INSIDE
%%
/* text */
"<inside>" { ECHO; BEGIN INSIDE; };
<INSIDE>"</inside>" { ECHO; BEGIN INITIAL; };
%%
int yywrap() { return(1); };
int main() {
printf("%s %d\n", FILE, LINE);
//yylex();
};

If flex is not already present on the system it is currently impossible to build it from git.
Therefore "You need the following tools to build flex from the maintainer's repository:" in README.md is no longer valid.

automake 1.15/autoconf 2.69/libtool 2.4.6/bison 3.0.4

/bin/bash ../build-aux/ylwrap scan.l .c scan.c -- /bin/bash /opt/kross/src/flex/build-aux/missing flex
/home/asavah/kross/src/flex/build-aux/missing: line 81: flex: command not found
WARNING: 'flex' is missing on your system.
         You should only need it if you modified a '.l' file.
         You may want to install the Fast Lexical Analyzer package:
         <http://flex.sourceforge.net/>
Makefile:1355: recipe for target 'scan.c' failed
make[3]: *** [scan.c] Error 127

Release tarball already contains pregenerated scan.c .

Copied from sourceforge #22

I have inherited a larger lex/yacc written compiler
which runs just fine compiled with classic lex on
solaris 2X. I want to port that program to linux but I
have some trouble with the flex.
I have isolated the section which causes me the problem.

The program below should just print abc but I am
getting bc and abc. Something is wrong with input/unput.

Am I doing something which is not allowed ?

--------------------

Linux baer 2.6.16.21-0.13-smp #1 SMP Mon Jul 17
17:22:44 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux

--------------------

flex -V
flex 2.5.31

--------------------

cat e.l

--------------------

%option noyywrap
%array
%%
"A" printf("a");
"B" printf("b");
"C" printf("c");
%%
main () {
yyin=stdin;
yyrestart(yyin);
unput(input());
yylex();
}

--------------------

flex -l e.l
cc -g lex.yy.c

--------------------

echo "ABC" | ./a.out
bc
abc

Hi!

Why latest version on sourceforce is 2.6.0 and Github has release 2.6.1? Is this 2.6.1 is a stable release?

Thanks.

When compiling the .cc file created by flex, I get the following warning:

Flex.cc:8730:23: warning: reference cannot be bound to dereferenced null pointer in well-defined C++ code; comparison may be assumed to always evaluate to false [-Wtautological-undefined-compare]
        b->yy_input_file = (&file == 0) ? NULL : file.rdbuf();

In short, there is an error in the code produced by flex.

https://github.com/westes/flex/blob/master/src/flex.skl#L1781 is raising sign-compare warnings with flex 2.6.0 on debian sid for me:

gcc-5 / gcc-6:

<stdout>: In function ‘yy_get_next_buffer’:
<stdout>:1113:44: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]

clang-3.8:

<stdout>:1113:44: warning: comparison of integers of different signs: 'int' and 'yy_size_t' (aka 'unsigned long') [-Wsign-compare]
        if ((int) ((yy_n_chars) + number_to_move) > YY_CURRENT_BUFFER_LVALUE->yy_buf_size) {

Conduct an audit and ensure that strings marked for translation through gettext follow the recommended best practices. I recall that some gettext-marked strings include technical terms in the string. Correct this as well as any other problems found with the translatable strings.

Hello,
There is currently no way to verify the complete integrity of releases. Please consider signing your releases so that package maintainers can verify the sources prior to packaging.

See here for how-to and complete details:
https://wiki.debian.org/Creating%20signed%20GitHub%20releases

Thank you

Hi all,

using flex I stumbled upon the following problem: yy_scan_buffer always calls yy_switch_to_buffer, which seems to make it impossible to use this call with yypush_buffer_state, like in the following example:
YY_BUFFER_STATE buffer_state = yy_scan_buffer(buffer.data, buffer.length);
yypush_buffer_state(buffer_state);

Do I miss something here? I circumvented this behavior with a custom build of flex, but that is not the cleanest solution. Would you mind adding an option for this?

Thanks in advance!

Hi guys, the LibreOffice project uses flex - many thanks for your great work.

We regularly get warnings about the unused parameter yyscanner, and yyunput is defined and not used.

Some time ago, a patch was proposed on the flex ML by Jesse Long to silence at least the parameter warnings:

http://sourceforge.net/p/flex/mailman/message/1198985/

Is there any technical reason this could not be implemented?

An example is:

[ CXX ] store/source/object.cxx
/home/chris/repos/LibreOffice/core/workdir/LexTarget/unoidl/source/sourceprovider-scanner.cxx:2438:13: warning: unused parameter ‘yyscanner’ [-Wunused-parameter]
static void yy_fatal_error (yyconst char* msg , yyscan_t yyscanner)
^
/home/chris/repos/LibreOffice/core/workdir/LexTarget/unoidl/source/sourceprovider-scanner.cxx:2782:7: warning: unused parameter ‘yyscanner’ [-Wunused-parameter]
void yyalloc (yy_size_t size , yyscan_t yyscanner)
^
/home/chris/repos/LibreOffice/core/workdir/LexTarget/unoidl/source/sourceprovider-scanner.cxx:2787:7: warning: unused parameter ‘yyscanner’ [-Wunused-parameter]
void *yyrealloc (void * ptr, yy_size_t size , yyscan_t yyscanner)
^
/home/chris/repos/LibreOffice/core/workdir/LexTarget/unoidl/source/sourceprovider-scanner.cxx:2799:6: warning: unused parameter ‘yyscanner’ [-Wunused-parameter]
void yyfree (void * ptr , yyscan_t yyscanner)
^
/home/chris/repos/LibreOffice/core/workdir/LexTarget/unoidl/source/sourceprovider-scanner.cxx:1952:17: warning: ‘void yyunput(int, char, yyscan_t)’ defined but not used [-Wunused-function]
static void yyunput (int c, register char * yy_bp , yyscan_t yyscanner)

Reposting the issue from the sourceforge https://sourceforge.net/p/flex/bugs/162

The attached test case has one exclusive state COMMENT. Some COMMENT rules there (obviously) can't be matched.
The problem is that addition of some rule for INITIAL state causes the warnings to disappear.

When the XXX rule is commented out, flex (correctly) prints these warnings:
testcase.l:17: warning, rule cannot be matched
testcase.l:18: warning, rule cannot be matched

When the XXX rule is present, flex (incorrectly) doesn't print these warnings.

Any rules in any other states shouldn't affect exclusive states. Each exclusive state should behave exactly as if it was the only state in the lex source.

flex-2.5.38_2 on FreeBSD-9.2

https://sourceforge.net/p/flex/bugs/162/attachment/testcase.l

Currently clang-based systems still see a flood of warnings from the 'register' keyword, that has been removed by my PR#5 10 months ago. Warnings come from the scanner itself, not from flex, so they are is many other projects.

Hi,

This bug was reported by a Debian user in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=729693

credit.l.txt

> flex credit.l
credit.l:2: unrecognized %option: line
> egrep option credit.l 
%option noyywrap
%option noline

Manoj

Hi,

This was reported by a Debian user in Bug 770161

In version 2.5.35 the type of yy_n_chars is int, while in 2.5.39 the type is yy_size_t. The yy_size_t is internal type mapped to size_t which is unsigned. The YY_INPUT macro contains following code:

#define YY_INPUT(buf,result,max_size) \
    errno=0; \
    while ( (result = read( fileno(yyin), (char *) buf, max_size )) < 0 ) \
    { \
            if( errno != EINTR) \

then the macro is then invoked:

YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
                    yyg->yy_n_chars, num_to_read );

So the code casts signed result of read() to unsigned and then compare if it's lower than 0.

Manoj

Here is how to reproduce:

credit.l.txt

flex -Cr credit.l
_> gcclint lex.yy.c
<>

lex.yy.c:1203:3: note: in expansion of macro ‘YY_INPUT’
   YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
   ^
lex.yy.c:781:20: warning: conversion to ‘yy_size_t {aka long unsigned int}’ from ‘ssize_t {aka long int}’ may change the sign of the result [-Wsign-conversion]
  while ( (result = read( fileno(yyin), (char *) buf, max_size )) < 0 ) \
                    ^
lex.yy.c:1203:3: note: in expansion of macro ‘YY_INPUT’
   YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
   ^
lex.yy.c:781:66: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits]
  while ( (result = read( fileno(yyin), (char *) buf, max_size )) < 0 ) \
                                                                  ^
lex.yy.c:1203:3: note: in expansion of macro ‘YY_INPUT’
   YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
   ^
lex.yy.c: In function ‘input’:
lex.yy.c:1363:23: warning: conversion to ‘yy_size_t {aka long unsigned int}’ from ‘long int’ may change the sign of the result [-Wsign-conversion]
    yy_size_t offset = (yy_c_buf_p) - (yytext_ptr);
                       ^

There are at least some strings that should be translated by gettext that are not. Any such strings should marked for translation so that a fuller localization of gettext is possible.

Hi,

This issue was reported by a debian user in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=512906

According to section 1.2 in the Bison manual (v 2.3):

"A terminal symbol that stands for a particular keyword in the language
should be named after that keyword converted to upper case."

However, flex defines a macro with the name BEGIN, which makes it
impossible to follow this convention if the language contains the
keyword `begin'.

I suggest renaming BEGIN to FLEX_BEGIN or YY_BEGIN.

Thanks,

Manoj

Hi,

This was reported by a Debian user in Bug 761250

The buf_append function will check if the current buf size needs
additional memory to append the new element(s).
n_alloc seems for how many ELEMENTS after append the new element(s),
so it is just a count number, and do not need to multiply with the
element size.

File: buf.c: http://sources.debian.net/src/flex/2.5.39-8/buf.c/#L245
241 /* May need to alloc more. /
242 if (n_elem + buf->nelts > buf->nmax) {
243
244 / exact amount needed... /
245 n_alloc = (n_elem + buf->nelts) * buf->elt_size;
246
247 / ...plus some extra */
248 if (((n_alloc * buf->elt_size) % 512) != 0
249 && buf->elt_size < 512)
250 n_alloc +=
251 (512 -
252 ((n_alloc * buf->elt_size) % 512)) /
253 buf->elt_size;

Check line 245, n_alloc should direct equals n_elem + buf->nelts, the
additional multiply with buf->elt_size should be a mistake, because in
line 248 and below, when checking the 512 boundaries, the n_alloc will
multiply with the buf->elt_size.

manoj

A lot of compiler warnings are generated by the current mess of variable types in flex. In general, this means two things:

• We haven't taken a wholistic approach to "how big can things get?" which is more relevant now that we're well into a 64bit era.
• We're careless re signed and unsigned comparisons.

To consider this issue resolved, we need:

• A lexer that compiles under some set of options like:

-Werror -Wtype-limits -Wconversion

cleanly. There are likely other flags that are relevant as well.

• An approach that accounts for the fact that the various integer types can be of varying sizes. In particular, we'd like to let things grow as much as we can without imposing any arbitrary limits that we don't have to.
• POSIX imposes things like yyleng should be an int. It's not so much that that is wrong (but it is), but that people (rightly) expect yyleng to be an int, so we need a way to work with that expectation. In an ideal world, we'd provide "here's the crappy thing you expect because the standard says so and we get that" as well as "here's the bigger better thing that lets you work up to the limits of what a lexer written in C can do".

Hello,

by default makeinfo is needed for documentation generation, but not checked by configure.

furthermore no make info ends p with the following error
'''
/local/gensoft2/src/flex/flex-2.6.1/build-aux/missing: line 81: makeinfo: command not found
WARNING: 'makeinfo' is missing on your system.
You should only need it if you modified a '.texi' file, or
any other file indirectly affecting the aspect of the manual.
You might want to install the Texinfo package:
http://www.gnu.org/software/texinfo/
The spurious makeinfo call might also be the consequence of
using a buggy 'make' (AIX, DU, IRIX), in which case you might
want to install GNU make:
http://www.gnu.org/software/make/
'''

even if

'''
c6builder:flex/flex-2.6.1 > make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu
'''

regards

Eric

For sure, this bug is in flex-2.5.37 and in looking at the source for the latest it's still there.
https://sourceforge.net/p/flex/bugs/180/